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Do charter middle schools improve students’ 
college outcomes? 


The charter school sector plays an important role in efforts to reform the education system and 
better serve the nation’s public school students. To help understand this role, more than a decade 
ago the National Evaluation of Charter Middle Schools examined how admission to about 30 
charter middle schools affected students’ achievement. On average, there were no significant 
effects. However, some charter middle schools in the study were successful in improving 
achievement, including those in urban areas and serving economically disadvantaged students — 
features of many charter schools today. This brief examines if the charter middle schools included 
in the National Evaluation affected students’ college enrollment and completion and the 
relationship between each school’s earlier achievement results and these important longer-term 
outcomes. 


Key findings include: 


e Being admitted to a charter middle school in the study did not affect students’ chances of 
enrolling in or completing college. 


e Individual charter middle schools’ success in improving students’ middle school achievement 
was not related to their success in improving students’ college enrollment and completion. 


Charter schools have played a key role in efforts to reform the public education system in the 
United States since the first ones opened in Minnesota in 1992. Charter schools are publicly financed, 
like traditional public schools, but are not governed by many of the same regulations such as those 
involving staffing, curriculum, and budget decisions. The charter school sector has grown 
substantially since 1992 and, as of the 2015—2016 school year, nearly 7,000 charter schools served 
about 2.8 million students across the United States (National Center for Education Statistics, 2018).! 


Given the number of students served by public charter schools, it is important for policymakers 
and parents to understand whether they are a good alternative to traditional public schools. This 
includes knowing whether they improve students’ outcomes in the short term (e.g., their effect on 
student achievement or academic learning). It also includes knowing whether they improve longer- 
term outcomes like college enrollment and completion, which are linked with students’ later career 
and earnings success (Deming, Yuchtman, Abulafi, Goldin, & Katz, 2016; Jepsen, Troske, & 
Coomes, 2014; Zimmerman, 2014). 


What is known about charter school effects on achievement and college outcomes? 


Most research studies that examine groups of charter schools, including both urban and non- 
urban charter schools, suggest that, on average, students who attend charter schools perform 
academically about as well as, but no better than, students at nearby traditional public schools (for 
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example, Center for Research on Educational Outcomes, 2013; Furgeson et al, 2012; Gleason, Clark, 
Tuttle, & Dwoyer, 2010; Zimmer, Gill, Booker, Lavertu, & Witte, 2012).* However, these effects on 
academic achievement can vary widely, depending on the groups of students served, the educational 
approaches used by the charter schools, and the nearby traditional public schools the students would 
otherwise have attended. Specifically, the most successful charter schools are those located in urban 
areas and those that serve large numbers of disadvantaged students (see Angrist, Pathak, and Walters, 
2013, and Gleason et al., 2010). 


Less is known about how charter schools affect students in the long run, and the few studies that 
have been conducted tell a mixed story. Some studies suggest attending charter schools does not affect 
students’ chances of enrolling in college (e.g., Angrist, Cohodes, Dynarski, Pathak, & Walters, 2016; 
Dobbie & Fryer, 2015), while others suggest attending charter schools does improve college 
outcomes (e.g., Sass, Zimmer, Gill, & Booker, 2016). Moreover, some evidence hints that the charter 
schools that improve students’ achievement in the short term are not necessarily the same ones that 
improve students’ educational attainment in the long term (see Dobbie and Fryer, 2015, and Sass et 
al., 2016).° 


To provide additional evidence on how charter schools affect students’ college enrollment and 
completion,+ Mathematica Policy Research used data from its previous study of charter middle 
schools, conducted for the U.S. Department of Education’s Institute of Education Sciences (IES), and 
collected new information on students’ long-term outcomes.” This brief addresses two questions: 


1. Does admission to a charter middle school affect students’ college enrollment and completion? 


2. Is charter middle schools’ success in improving students’ middle school achievement related to 
their success in improving students’ college enrollment and completion? 


How was the study conducted? 


The research team for the original IES study recruited 36 charter middle schools in 15 states that 
met the study requirements and agreed to participate.° For a charter school to participate in the study, 
it had to have more applicants than spots available and admit students by lottery, so that outcomes for 
lottery winners and losers could be compared. This limited the study sample to charter middle schools 
that were in relatively high demand (fewer than 18 percent of the 492 charter schools nationwide that 
were considered for this study) (Gleason et al., 2010, p. 6). 


In other respects, the charter middle schools in the study were a diverse group. They were spread 
across every region of the United States (Gleason et al., 2010, p.7) and had the following 
characteristics: 


e About one-third of participating schools (32 percent) were located in large urban areas, and 
the rest were in smaller cities, suburbs, or rural areas (Figure 1). 

e The average study school had an enrollment of 387 students and had been operating for seven 
years (Gleason et al., 2010, pages 26-27). 

e Most (55 percent) were low-poverty schools where less than one-quarter of students were 
eligible for free or reduced-price school meals. A minority (10 percent) had high poverty 
levels, with more than three-quarters of students eligible for free or reduced-price meals 
(Figure 2). 

e About half of the schools (52 percent) had racially and ethnically diverse student populations. 
The other schools were less diverse, with mostly white (29 percent of schools) or mostly 
minority (19 percent of schools) student populations (Figure 2). 
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Figure 1. The study included a mix of urban and non-urban charter middle schools 


Percentage of schools 


32% 


= Urban (10 charter schools) 


Non-urban (21 charter schools) 


68% 


Source: Common Core of Data. 


Note: This figure represents the 31 charter middle school sites used in the main analysis. Most charter schools are their 
own site, but in five cases two charter schools were combined into a single charter school site (see Appendix A). 


Figure 2. Study charter schools served diverse populations of students 


Percentage of schools 
19% 


29% 
= Mostly white 
( <25% minority students) 


= Mix of white and non-white 
(25-75% minority students) 


Mostly non-white 
(>75% minority students) 


10% 


= Low poverty 
( <25% of students) 


= Medium poverty 


35% 55% (25-75% of students) 


High poverty 
( >75% of students) 


Source: Data from principal surveys and administrative records, collected by Mathematica Policy Research. 
Note: This figure represents the 31 charter middle school sites used in the main analysis. 
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A total of 2,904 students participated in the original study, including 1,744 lottery winners and 
1,160 lottery losers. Of these, college data were obtained from the National Student Clearinghouse 
for 2,873 students (1,723 lottery winners and 1,150 lottery losers) who are used to examine the effect 
of admission’ to charter middle schools in the study on college enrollment and completion.® The 
analysis comparing the impacts on college outcomes with the impacts on middle school achievement 
is limited to the smaller number of students who also had math and reading test scores available in 
the original study (1,295 lottery winners and 883 lottery losers) (See Appendix A, Figure A.1.). By 
December 2017, when college outcome data were obtained, most students in the original study (82 
percent) would have had five or six academic years during which they could have enrolled in college, 
if they graduated from high school on a typical schedule (see Appendix A, Table A.3).? Data on 
student and school characteristics and student middle school math and reading test scores came from 
the original study (See Appendix A, Table A.5). 


This update to the earlier study examined two primary college outcomes: (1) whether a student 
ever enrolled in any postsecondary institution, and (2) whether a student earned a degree (bachelor’s, 
associate’s, or certificate) or was still enrolled as of December 2017. To better understand the primary 
study results, this study also explored how admission to a charter middle school affected additional 
outcomes related to college enrollment and completion (see Table 1). 


Table 1. Exploratory outcomes 


Exploratory outcomes related to college enrollment de) (e)e- 1eelameolbinere)gnt-\-Mac)t-1k-\e mcemexe) |(<1--Merelan]e)(-14() 4) 
Ever enrolled in postsecondary institution Graduated from postsecondary institution 

2-year college Associate’s degree (2-year college) 

4-year college Bachelor’s degree (4-year college) 


Graduated with bachelor’s degree within 4 years of 


Type of institution where currently or most recently enrolled projected high school graduation 


Public 

Private nonprofit Graduated or currently enrolled 

Private for-profit Graduated with associate’s degree or currently enrolled in 2- 
_ ae year college 

Selectivity of postsecondary institution Graduated with bachelor’s degree or currently enrolled in 4- 

Highly selective (1-25% acceptance rate) year college 

Selective (26-50% acceptance rate) 

Somewhat selective (51-75% acceptance rate) Number of years completed 


At 2-year college 
ees At 4-year college 
Dual enrollment participation At any postsecondary institution 


Less selective (76-100% acceptance rate) 


Note: When comparisons between lottery winners and losers are made for a large number of outcomes, as proposed for the 
exploratory analysis, there is an increased likelihood of finding some effects - positive or negative - that are actually 
due to chance. 


To measure the effects of admission to a charter middle school, the outcomes of lottery winners 
and losers were compared. Because lottery winners were selected at random, lottery winners and 
lottery losers were similar in all respects at the start of the study, on average, and any differences in 
their outcomes could be attributed to the effect of being admitted to a study charter school. Ifa larger 
percentage of lottery winners enrolled in college, for example, the results would indicate that 
admission to a charter middle school in the study increased students’ college enrollment. !° Because 
22 percent of students who won the lottery opted not to attend the charter school that offered them 
admission, the study also estimated the effects of attending a charter middle school." 
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Finding 1: Being admitted to a charter middle school did not affect students’ college 
enrollment or completion 


A key mission of most middle schools is to improve the educational success of students who 
enroll. If charter schools are better able than traditional public schools to meet this goal — through 
raising academic achievement or imparting other skills and behaviors that matter later — their students 
may enter high school better prepared to take higher-level classes, study hard, and have higher 
expectations for themselves. These students may, in turn, graduate high school better prepared for 
college. By shifting the educational trajectory of these students early on, charter middle schools could 
set them on a path to higher rates of college enrollment and completion, regardless of where they 
attended high school. The study investigated this hypothesis. 


One indicator of a charter school’s long-term success is whether its former students enroll in 
college. However, many students who enroll in college never complete the programs they enter. For 
instance, nationally, only 29 percent who entered two-year colleges in 2012 had graduated with an 
associate’s degree within three years, and only 59 percent who enrolled in four-year colleges in 2009 
had graduated with a bachelor’s degree within six years (McFarland et al., 2017). Charter schools 
may be more (or less) likely to give students the kinds of skills that allow them to persist in and 
complete their college program. Therefore, the study also examined whether charter middle schools 
affected students’ chances of earning a degree (bachelor’s degree, associate’s degree, or certificate). !” 


’ 


Being offered admission to or attending a charter middle school did not affect students 
chances of enrolling in college. On average, 69 percent of both lottery winners and lottery losers 
enrolled in some type of college during the study period (Figure 3). This was the rate of college 
enrollment among 2015 high school graduates nationally (McFarland et al., 2017).'° There was also 
no effect from attending a charter middle school (rather than just being offered admission) on college 
enrollment. (Detailed results can be found in Appendix B, Table B.1.) 


These findings could mask other types of enrollment-related effects that could have 
consequences for students. For example, if charter schools create higher expectations for students 
than traditional public schools, students admitted to charter schools may be more likely to attend four- 
year rather than two-year colleges (Angrist et al., 2016, and Dobbie and Fryer, 2015), less likely to 
attend private for-profit institutions, and more likely to attend more selective colleges. 


However, the study did not find evidence that being offered admission to or attending a charter 
middle school in the study affected the types of colleges students attended: 


e Students offered admission to charter middle schools in the study were about as likely to attend 
four-year institutions and two-year institutions as those not offered admission (see Figure 3).'4 
There were also no effects on the number of years that students attended college, either overall 
or in four-year versus two-year schools. 


e There was no difference between lottery winners and losers in enrollment at public, private 
nonprofit, or private for-profit colleges. Among lottery winners, 55 percent enrolled in a 
public college, 13 percent in a private nonprofit college, and 1 percent in a private for-profit 
college. Lottery losers enrolled in these types of colleges at similar rates: 54 percent enrolled 
in a public college, 14 percent in a private nonprofit college, and 1 percent in a private for- 
profit college. 


e Being offered admission to a charter middle school in the study did not affect the selectivity 
of colleges that students attended. For example, 16 percent of lottery winners and 15 percent 
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of lottery losers attended highly selective (1 to 25 percent acceptance rate) or selective (26 to 
50 percent acceptance rate) colleges. !> 


Students offered admission to a charter middle school were no more or less likely to earn 
college credit while in high school (commonly known as “dual enrollment”), which studies 
suggest may lead to higher rates of college enrollment (An, 2013; U.S. Department of 
Education, 2017b). About 9 percent of students offered admission earned college credits while 
in high school compared to 10 percent of those who were not offered admission. !° 


Figure 3. Admission to a study charter school did not affect students’ college enrollment 


Percentage of students 


Note: 


70 


60 


50 


40 


30 


20 
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80 


69 69 


55 52 


32 35 


Ever enrolled in college Ever enrolled ina Ever enrolled ina 
4-year college 2-year college 


= Charter school lottery winners Charter school lottery losers 


The percentages reported include 2,873 students, 99 percent of the 2,904 students in the original study. In both the 
lottery winner and lottery loser groups, 3 percent of students who attended a postsecondary institution opted out of 
sharing their data. (The National Student Clearinghouse [NSC] reports a count of the number of students in each data 
request file who are found in NSC records but who opted out of sharing data.) The study team had no way to know 
whether individual students opted out of sharing data or were never enrolled in a postsecondary institution. Therefore, 
for both the lottery winners and lottery users, this figure underestimates overall enrollment by approximately 3 
percentage points, but the estimated difference between the two groups was not affected. The percentages were 
generated from the study’s regression model, as described in Appendix A. The percentages represent the average 
across all 31 charter school sites in the study. Each site estimate was based on a weighted analysis where the weight 
reflected each student’s probability of assignment to the lottery winner or lottery loser group. 

The differences between lottery winners and lottery losers are not statistically significant at the .05 level, two-tailed 
test. The estimated shift from enrollment in two-year to enrollment in four-year colleges is not significantly different 
from zero at the .05 level, two-tailed test, but is significant at the 0.10 level (p = 0.097). 


Being offered admission to a charter middle school did not affect students’ chances of 


obtaining a degree or still being enrolled in college at the end of the study period. On average, 48 
percent of lottery winners and 47 percent of lottery losers had a degree or were still enrolled as of 
December 2017 (Figure 4). Attending a charter middle school in the study also did not affect students’ 
chances of obtaining a degree or continuing to be enrolled in college. Similar percentages of lottery 
winners and lottery losers had obtained two-year associate’s degrees and four-year bachelor’s 
degrees, or were still enrolled, by the end of the study period (see Figure 4). No differences were 
found for other exploratory outcomes related to college completion. Detailed results for college 
completion outcomes can be found in Appendix B, Table B.1. 
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Figure 4. Admission to a study charter school did not affect students’ college completion 


Percentage of students 
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Has degree or Has a bachelor's degree or Has an associate's degree or 
currently enrolled in college currently enrolled in 4-year college currently enrolled in 2-year college 
= Charter school lottery winners Charter school lottery losers 


This analysis includes 2,873 students, 99 percent of the 2,904 students in the original study. In both the lottery winner 
and lottery loser groups, 3 percent of students who attended a postsecondary institution opted out of sharing their 
data. (The National Student Clearinghouse [NSC] reports a count of the number of students in each data request file 
who are found in NSC records but who opted out of sharing data.) The study team had no way to know whether 
individual students opted out of sharing data or were never enrolled in a postsecondary institution. Therefore, for both 
the lottery winners and lottery users, this figure underestimates overall enrollment by approximately 3 percentage 
points, but the estimated difference between the two groups was not affected. The percentages were generated from 
the study’s regression model, as described in Appendix A. The percentages represent the average across all 31 charter 
school sites in the study. Each site estimate was based on a weighted analysis, where the weight reflected each 
student’s probability of assignment to the lottery winner or lottery loser group. 


For the ever-enrolled outcomes, students may be represented in both categories (both four-year and two-year colleges) 
if they attended both a two-year and a four-year college at some point between 2009 and 2017. For the completion 
outcomes, students who had obtained an associate’s degree were not included in the two-year outcome if they later 
obtained a bachelor’s degree. 


There was no evidence that being offered admission to a charter middle school in the study 
had different effects on college enrollment or completion for students with different characteristics, 
such as their economic backgrounds, race/ethnicities, or prior achievement levels. It is important 
to understand whether charter schools could help reduce differential rates of college enrollment and 
improve college outcomes for students who are from racial or ethnic subgroups, are from lower 
socioeconomic backgrounds, or have lower baseline (pre-lottery) achievement scores. For each of 
these groups of students, the pattern of findings was similar to the pattern of findings for all students 
in the study, as described above (see Appendix B, Tables B.2 and B.3). 
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Finding 2: Individual charter middle schools’ effects on college enrollment and 
completion were not related to their effects on students’ middle school achievement 


Attending and completing college has long been considered a gateway to career advancement 
and higher earnings, with academic achievement in pre-college grades a stepping stone along this 
path (Card, 1999; Chetty, Friedman, Saez, Turner, & Yagan, 2017). If academic achievement in 
middle school is critical to students’ long-term success, we might expect that the individual schools 
that were most successful in improving students’ middle school test scores would also be most 
successful in improving their long-term outcomes. However, if other factors or areas of student 
development are most important, this may not be the case. For example, students’ experiences in high 
school, regardless of their middle school achievement, may be most important in determining long- 
term outcomes. 


Federal and state policies promoting school improvement often presume a link between 
improving students’ academic achievement and their later success in life, including in postsecondary 
education.!’ Yet several recent studies are beginning to question that assumption, as noted above. 
Therefore, this study explored the extent to which the short-term (achievement) and long-term 
(college) success of schools and students were related. 


There was little evidence that a charter middle school’s effects on middle school test scores 
were related to the school’s effects on college enrollment and completion. Among charter schools 
in this study, for example, those in urban areas and those serving more disadvantaged students had 
more positive effects on students’ middle school achievement (Gleason et al., 2010). However, these 
same subgroups of charter schools did not have more positive effects on students’ long-term college 
outcomes (Appendix B, Table B.4 and Figures B.1 and B.2). 


More generally, schools that were successful in improving middle school achievement were not 
consistently more successful than others in boosting students’ college enrollment or completion 
(Figure 5). In the figure, each point represents one study charter school. This figure shows whether 
each study charter school had positive (or negative) effects on both middle school achievement and 
college enrollment (top graph) or completion (bottom graph), as well as how large or small those 
effects were. For example, a point in the top right quadrant represents a school where lottery winners 
had better outcomes than lottery losers on both middle school achievement and college enrollment 
(top) or college completion (bottom). In each case, the schools are scattered throughout the graph 
without any clear patterns, which indicates that there is not a consistent relationship between the 
effects that schools have on students’ middle school achievement and college enrollment or between 
the effects on middle school achievement and college completion. 


Looking across all charter middle schools in the study, there was no statistically significant 
relationship between how schools affected achievement and how they affected college outcomes (see 
Appendix B, Tables B.6 and B.8). For students in the study, middle school achievement was related 
to college outcomes, but the strength of that relationship was modest (see Appendix B, Tables B.5 
and B.7). '® These results suggest that factors other than middle school achievement may have been 
more influential in promoting college enrollment and completion among the students in this study. 
For example, students’ motivation or persistence may have contributed to their success in college. 
Alternatively, the students’ experiences in high school may have outweighed their middle school test 
scores in determining whether they entered and completed college. 
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Figure 5. Charter middle schools’ effects on middle school test scores were not related to schools’ 
effects on college enrollment and completion 


+ 


Positive T 


effect on 

college 
enrollment 
(proportions) 


Negative 
effect on 
college 
enrollment 
(proportions) 


—> + 


Positive 
effect on 
college 
completion 
(proportions) 


Negative 
effect on 
college 
completion 
(proportions) 


Note: 


0.4 
e 
0.2 e e . 
e 
e ee © 
e ° 
e e 
0 e° e ° 
e 
e 
. ee 
e = e 
-0.2 
e 
e 
-0.4 
-0.6 -0.4 -0.2 6) 0.2 0.4 0.6 
Negative effect on middle Positive effect on middle 
school achievement school achievement 
(standard deviations) (standard deviations) 
0.4 
e 
e 
7 e 
0.2 . 
e 
e 
e 
bd e 
e 
e ee 
0 e 
° e 
° e 
° e 
°° 
-0.2 ee e 
e 
-0.4 
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 


Negative effect on middle 
school achievement 
(standard deviations) 


Positive effect on middle 
school achievement 
(standard deviations) 


This figure shows whether each study school had positive (or negative) effects on middle school achievement and 


college enrollment/completion; that is, where lottery winners had better (or worse) outcomes than lottery losers. Each 
point represents one school. Students’ middle school math and English language arts state test scores were 
standardized to allow for comparability across the 15 states represented in the study, and the middle school 
achievement effects are reported in standard deviations of student achievement. The college enrollment and 
completion effects are reported in proportions, which represents the fraction of the sample from each school who 
enrolled in college (top graph) and completed college (bottom graph). The sample includes the 1,919 students that 
had middle school test score data and postsecondary outcome data in 28 charter school sites. 
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Conclusion 


There is little evidence that admission to the charter middle schools in this study affected 
students’ college enrollment or completion, on average. This finding is similar to the earlier study of 
this same group of schools, which showed that they did not affect students’ middle school 
achievement, on average (Gleason et al., 2010). However, although urban charters and those serving 
disadvantaged students positively affected their students’ achievement in middle school, they were 
not equally successful in raising those students’ enrollment in and completion of college. Nor did the 
negative impacts on middle school achievement found for non-urban charters and those serving more 
advantaged students lead to negative impacts on long-term college outcomes. 


The study had a few important limitations. First, little is known about what happened to these 
students after middle school and before they had the chance to enter college, particularly whether 
students who attended the charter middle schools in the study also attended charter high schools. 
Because high schools may do more to directly prepare students for college than middle schools, 
students who attended both charter middle and charter high schools could have different college 
outcomes than students who returned to a traditional public school after middle school. A study on 
the long-term effects of charter high schools might show different results. Second, the study focused 
on charter middle schools that were operating in the 2005—2006 and 2006-2007 school years, and 
charter school practices that could influence students’ long-term success might have changed since 
then. Finally, this study focused on a limited number of charter middle schools, and all of the schools 
in the study held admissions lotteries. The results might have been different if the study included a 
different or larger set of charter schools. This limitation is especially important in interpreting the 
findings related to groups of charter schools with similar characteristics (e.g., schools in urban areas). 


The study’s results point to the importance of learning more about how charter schools affect the 
long-term outcomes of their students. Increasingly, many policymakers and parents may care more 
about charter schools’ effects on students’ longer-term success in college and beyond than about 
interim indicators like test scores (U.S. Department of Education, 2017a). 


NCEE EVALUATION BRIEF 


DO CHARTER MIDDLE SCHOOLS IMPROVE STUDENTS’ COLLEGE OUTCOMES? 11 


References 


An, B. P. (2013). The impact of dual enrollment on college degree attainment: Do low-SES students 
benefit? Educational Evaluation and Policy Analysis, 35(1), 57-75. 


Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. R. (2016). Stand and 
deliver: Effects of Boston's charter high schools on college preparation, entry, and choice. 
Journal of Labor Economics, 34(2), 275-318. 


Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using 
instrumental variables. Journal of the American Statistical Association, 91(434), 444-455. 


Angrist, J. D., Pathak, P. A., & Walters, C. R. (2013). Explaining charter school 
effectiveness. American Economic Journal: Applied Economics, 5(4), 1-27. 


Card, D. (1999). The causal effect of education on earnings. In Handbook of labor economics, 3(A), 
1801-1863. Elsevier. 


Center for Research on Educational Outcomes (2013). National charter school study: 2013. 
Stanford, CA: Center for Research on Educational Outcomes, Stanford University. 


Chetty, R., Friedman, J. N., Saez, E., Turner, N., & Yagan, D. (2017). Mobility report cards: The 
role of colleges in intergenerational mobility (No. w23618). National Bureau of Economic 
Research. 


David, R., & Hesla, K. (March 2018). Estimated public charter school enrollment, 2017-2018. 
National Alliance for Public Charter Schools. Retrieved from 


https://www.publiccharters.org/sites/default/files/documents/2018- 
03/FINAL%20Estimated™%20Public%20Charter%20School™%20Enrollment%2C%202017- 


18.pdf 


Deming, D. J., Yuchtman, N., Abulafi, A., Goldin, C, & Katz, L. F. (2016). The value of 
postsecondary credentials in the labor market: An experimental study. American Economic 
Review, 106(3), 778-806. 


Dobbie, W., & Fryer, R. G., Jr. (2015). The medium-term impacts of high-achieving charter 
schools. Journal of Political Economy, 123(5), 985-1037. 


Dobbie, W., & Fryer, R. G., Jr. (2016). Charter schools and labor market outcomes (NBER 
Working Paper No. 22502). Cambridge, MA: National Bureau of Economic Research. 


Dundar, A., & Shapiro, D. (2016, May). The National Student Clearinghouse as an integral part of 
the national postsecondary data infrastructure. Herndon, VA: National Student Clearinghouse 
Research Center. 


Dynarski, S. M., Hemelt, S. W., & Hyman, J. M. (2015). The missing manual: Using National 
Student Clearinghouse data to track postsecondary outcomes. Educational Evaluation and 
Policy Analysis, 37(1S), 53S—79S. 


NCEE EVALUATION BRIEF 


12 DO CHARTER MIDDLE SCHOOLS IMPROVE STUDENTS’ COLLEGE OUTCOMES? 


Furgeson, J., Gill, B., Haimson, J., Killewald, A., McCullough, M., Nichols-Barrer, IL, ... Lake, R. 
(2012). Charter-school management organizations: Diverse strategies and diverse student 
impacts. Princeton, NJ: Mathematica Policy Research. 


Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school 
impacts: Final report (NCEE 2010-4029). Washington, DC: National Center for Education 
Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 
Education. 


Jepsen, C., Troske, K., & Coomes, P. The labor-market returns to community college degrees, 
diplomas, and certificates. Journal of Labor Economics, 32(1), 95-121. 


McFarland, J., Hussar, B., de Brey, C., Snyder, T., Wang, X., Wilkinson-Flicker, S., ... Hinz, S. 
(2017). The condition of education 2017 (NCES 2017-144). Washington, DC: National Center 
for Education Statistics, U.S. Department of Education. Retrieved March 9, 2018, from 
https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017144 


McFarland, J., Hussar, B., Wang, X., Zhang, J., Wang, K., Rathbun, A., ... Bullock Mann, F. 
(2018). The condition of education 2018 (NCES 2018-144). Washington, DC: National Center 
for Education Statistics, U.S. Department of Education. Retrieved October 30, 2018, from 
https://nces.ed.gov/pubs2018/2018144.pdf 


National Alliance for Public Charter Schools. (2017). Estimated charter public school enrollment, 
2016-17. Retrieved November 30, 2017, from 


http://www.publiccharters.org/publications/estimated-charter-public-school-enrollment-2016- 
Te 


Sass, T. R., Zimmer, R. W., Gill, B. P., & Booker, T. K. (2016). Charter high schools’ effects on 
long-term attainment and earnings. Journal of Policy Analysis and Management, 35(3), 683— 
706. 


Shaw, E. J., Marini, J. P., Beard, J., Shmueli, D., Young, L., & Ng, H. (2016). The redesigned 
SAT® pilot predictive validity study: A first look. The College Board. Retrieved March 8, 2018, 
from https://collegereadiness.collegeboard.org/pdf/redesigned-sat-pilot-predictive-validity- 


study-first-look.pdf 


Thomas, N., Marken, S., Gray, L., & Lewis, L. (2013). Dual credit and exam-based courses in U.S. 
public high schools: 2010-11 (NCES 2013-001). Washington, DC: U.S. Department of 
Education., National Center for Education Statistics. Retrieved May 8, 2018, from 


http://nces.ed.gov/pubsearch 


Unterman, R., Bloom, D., Byndloss, D. C., & Terwelp, E. (2016). Going away to school: An 
evaluation of SEED DC. New York: MDRC. 


U.S. Department of Education. (2017, January). Every Student Succeeds Act high school graduation 
rate non-regulatory guidance. Retrieved May 8, 2018, from 
https://www2.ed.gov/policy/elsec/leg/essa/essagradrateguidance.pdf 


NCEE EVALUATION BRIEF 


DO CHARTER MIDDLE SCHOOLS IMPROVE STUDENTS’ COLLEGE OUTCOMES? 13 


U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2017, 
February). Transition to College intervention report: Dual enrollment programs. Retrieved 
May 8, 2018, from https://whatworks.ed.gov 


U.S. Department of Education, National Center for Education Statistics, Common Core of Data. 
(2015). Public Elementary/Secondary School Universe Survey, 1999-2000 through 2013- 
14. Retrieved May 8, 2018, from 
https://nces.ed.gov/programs/digest/d15/tables/dt15_216.30.asp 


Zimmer, R., Gill, B., Booker, K., Lavertu, S., & Witte, J. (2012). Examining charter school 
achievement effects across seven states. Economics of Education Review, 31, 213-224. 


Zimmerman, S.D. (2014). The returns to college admission for academically marginal students. 
Journal of Labor Economics, 32(4), 711-754. 


NCEE EVALUATION BRIEF 


14 DO CHARTER MIDDLE SCHOOLS IMPROVE STUDENTS’ COLLEGE OUTCOMES? 


APPENDIX A: STUDY DESIGN AND ANALYTIC APPROACH 


This appendix provides additional details on the study design and analytic approach used to 
estimate the effect of charter middle school admission on students’ college enrollment and 
completion. !” 


Study design 


The design for this study is based on the design of the original Institute of Education Sciences 
(IES) study of charter middle schools (Gleason et al., 2010), and much of the data analyzed in this 
study were collected as part of the original study. This section provides key information about the 
study design, including the schools and students in the study, the charter schools’ admissions lotteries, 
data sources, analysis sample, and outcome measures. Additional details are provided in the original 
study report (Gleason et al., 2010). 


Schools and students in the study 


Characteristics of schools in the study. The team for the original IES study recruited charter 
middle schools from across the United States. A total of 36 charter middle schools in 15 states agreed 
to participate and met the following study requirements: (1) they had entry grades from grade 4 to 
grade 7, (2) they had been in operation for at least two years before the start of the study (allowing 
the study to focus on more established and stable schools), and (3) they had more students who applied 
for admission in the 2005—2006 and/or 2006—2007 school years than they could accommodate and 
used lotteries to randomly select students for admission. 


Because of the participation requirements, charter middle schools in the study differed in some 
respects from the average charter middle school in operation in the United States at the time of the 
study. 7° On average, participating schools had been operating longer, and served higher 
socioeconomic status students, fewer minority students, and higher performing students than charters 
nationwide (Table A.1). However, the sample schools were similar to the average charter middle 
school in other ways, including enrollment and location. (See Gleason et al., 2010, pages 25—27, for 
additional details.) 


Table A.1. Characteristics of study charter schools, compared to non-study charter schools 
8) 7) )) 


Study charter Other charter Difference 
Outcome For alele) fc) FYor gofe) =) 
Years of operation (mean) 7.0 5.9 1.2* 
Located in large urban area (percentage) 36 41 4 
Enrollment (mean) 
Total enrollment 387 298 90 
Student-teacher ratio 14.6 16.7 “11 
Student characteristics (percentage) 
Eligible for free or reduced-price meals 44 62 -18** 
Scored proficient on state math assessment 66 51 15** 
Belong to racial or ethnic minority 47 62 15* 
Learning disability and/or IEP 12 12 0 
Sample size—Characteristics based on principal survey or Common 36 434 
Core of Data 
Sample size—School test scores 36 380 


Source: Gleason etal. (2010), Table Ill.1, pages 26-27. 

*Difference significantly different from zero at the .05 level, two-tailed test. 
** Difference significantly different from zero at the .01 level, two-tailed test. 
IEP = individualized education program. 
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Characteristics of students in the study. A total of 2,904 students applied for admission to one 
of the study charter middle schools, participated in the school’s lottery, and provided parental consent 
to participate in the study. Students who were exempt from the lottery and automatically admitted to 
the school (including siblings of currently enrolled students and children of staff or board members) 
were not included in the study sample. (See Gleason et al., 2010, pages A-7 and A-8 for additional 
information.) 


Most of the students who applied to the study charter middle schools met their state’s proficiency 
standards for reading (71 percent) and math (66 percent) in the year that they participated in the lottery 
(Gleason et al., 2010). The student applicants were diverse: 53 percent were white, 13 percent were 
black, and 27 percent were Hispanic. In addition, one-third of the students qualified for free or 
reduced-price meals, 16 percent had an individualized education program (IEP), and 9 percent were 
English language learners or limited English proficient (Table A.2). 


Charter school lotteries 


The charter schools held their lotteries in the winter or spring before the 2005—2006 and 2006-— 
2007 school years, and most lotteries were public events. After the lotteries were conducted and initial 
lottery winners were offered admission, study charter schools continued to admit applicants from a 
randomly ordered waiting list as space became available. Applicants who were offered admission to 
the school based on the initial lottery results or who were offered admission from the waiting list 
formed the “lottery winners” group. Those who participated in the lottery and were not offered 
admission immediately after the initial lottery or from the waiting list formed the “lottery losers” 
group.”! Among the 2,904 students in the full sample, 1,744 were lottery winners and 1,160 were 
lottery losers. 


To ensure that study schools conducted admissions lotteries and admitted students in ways 
consistent with the principles of random assignment, members of the original study team attended 
nearly all of these lotteries in person and monitored the wait lists (see Gleason et al., 2010, pages A- 
12 to A-15 for more detail about the lotteries). As expected with random assignment, the lottery 
winners and losers had similar characteristics at the time of the lotteries. Among the 50 characteristics 
examined, there were statistically significant differences between the two groups for only 5 of them. 
Only two of the significant differences were larger than 3 percentage points. For instance, lottery 
winners were less likely to have families who earned more than 300 percent of the poverty level and 
more likely to have attended public school at the time of the lottery than lottery losers (Table A.2). 


The outcome of the lottery determined whether students in the study were offered admission to 
study charter schools, but did not guarantee that they would actually attend those schools. Conversely, 
some lottery losers might have ultimately attended study charter schools or some other charter 
schools. However, the majority of lottery winners attended charter schools, whereas most lottery 
losers attended traditional public schools. Of the lottery winners, 78 percent attended a study charter 
school. Of the lottery losers, 78 percent attended a traditional public school. Small proportions of 
students in each group attended non-study charter schools, private schools, or were home schooled 
(Gleason et al., 2010, page 20). 


NCEE EVALUATION BRIEF 


16 DO CHARTER MIDDLE SCHOOLS IMPROVE STUDENTS’ COLLEGE OUTCOMES? 


Table A.2. Characteristics of lottery winners and losers in the analysis, at the time of the lottery (baseline 
characteristics) 


(1) 7) ()) ) 
Mean, lottery Mean, lottery Mean, lottery Difference 


winners and winners losers 
Outcome losers 


Reading achievement 


Baseline reading score (z-score units) 0.40 0.39 0.41 -0.02 
Baseline reading proficiency (proportions) 
Advanced 0.26 0.25 0.26 -0.01 
Proficient or higher 0.70 0.70 0.74 -0.02 
Partially proficient or higher 0.92 0.91 0.92 -0.02 
Pre-baseline reading score (z-score units) 0.44 0.46 0.414 0.06 
Pre-baseline reading proficiency (proportions) 
Advanced 0.28 0.27 0.28 -0.01 
Proficient or higher 0.69 0.66 0.714 -0.04~ 
Partially proficient or higher 0.91 0.90 0.93 -0.02 
Math achievement 
Baseline math score (z-score units) 0.41 0.41 0.41 0.00 
Baseline math proficiency (proportions) 
Advanced 0.29 0.30 0.29 0.014 
Proficient or higher 0.66 0.67 0.65 0.02 
Partially proficient or higher 0.89 0.89 0.88 0.00 
Pre-baseline math score (z-score units) 0.42 0.48 0.36 0.12~ 
Pre-baseline math proficiency (proportions) 
Advanced 0.27 0.25 0.27 -0.02 
Proficient or higher 0.64 0.63 0.65 -0.02 
Partially proficient or higher 0.88 0.89 0.87 0.02 
Disciplinary measures 
Number of days absent in baseline school year 6.24 6.20 6.23 -0.03 
Student suspended in baseline school year 0.04 0.04 0.04 0.00 
(proportion) 
Demographic characteristics 
White, non-Hispanic (proportion) 0.53 0.54 0.52 0.03 
Black, non-Hispanic (proportion) 0.13 0.13 0.13 -0.01 
Other race, non-Hispanic (proportion) 0.07 0.06 0.08 -0.02~ 
Hispanic (proportion) 0.27 0.27 0.27 0.00 
Male (proportion) 0.47 0.47 0.47 0.00 
Age at start of school year 11.46 11.48 11.44 0.04 ~ 
Young for grade (proportion) 0.01 0.01 0.02 -0.01 
Old for grade (proportion) 0.10 0.10 0.09 0.01 
IEP status (proportion) 0.16 0.17 0.16 0.01 
Limited English proficiency/ELL (proportion) 0.09 0.10 0.07 0.02* 


Family characteristics (proportions) 
Income-to-poverty ratio 


O to 100 percent 0.13 0.14 0.14 0.03~ 
100 to 200 percent 0.20 0.21 0.19 0.02 
200 to 300 percent 0.17 0.17 0.17 0.00 
Greater than 300 percent 0.50 0.47 0.53 -0.05* 
Two-parent family 0.77 0.77 0.78 -0.01 
Not two-parent family, but more than one adult 0.04 0.05 0.04 0.01 
English main language spoken at home 0.90 0.89 0.91 -0.01 
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(1) 7) i) (4) 
Mean, lottery Mean, lottery Mean, lottery Difference 


winners and winners losers 
Outcome losers 


Mother’s education 


High school or less 0.24 0.25 0.24 0.014 
Some college 0.33 0.33 0.34 0.00 
College degree or more 0.42 0.42 0.43 -0.01 
Born in United States 0.93 0.93 0.93 0.00 
Received TANF or food stamps in past year 0.06 0.08 0.05 0.02* 
Free or reduced-price meal-eligible 0.35 0.35 0.34 0.01 
Number of children in household 
One 0.23 0.23 0.23 0.00 
Two 0.46 0.46 0.45 0.01 
Three or more 0.32 0.31 0.32 -0.01 
Baseline school enrollment (proportions) 
Enrolled in charter school 0.05 0.04 0.07 -0.03* 
Enrolled in private school 0.10 0.08 0.14 -0.03~ 
Enrolled in public school 0.83 0.86 0.81 0.05* 
Home-schooled 0.01 0.02 0.014 0.01 
School type unknown 0.03 0.02 0.03 -0.01 
Changed schools midyear 0.01 0.01 0.01 0.00 
Number of students 2,873 1,723 1,150 
Number of sites 31 
Note: Sample includes students in the brief’s impact analysis sample presented in Finding 1. 


Means are calculated at the charter school site level and averaged across all 31 sites. Sample sizes vary by 
characteristic because some students have missing data for some characteristics. The reported sample size is the 
maximum number of students. This table presents only non-missing data (no imputed values are included). 


*Difference significantly different from zero at the .05 level, two-tailed test. 
**Difference significantly different from zero at the .01 level, two-tailed test. 
~Difference significantly different from zero at the .10 level, two-tailed test. 
ELL = English language learner; IEP = individualized education program; TANF = Temporary Assistance for Needy Families. 


Data sources 


To measure how the charter middle schools in the study affected students’ college enrollment 
and completion, and to examine the relationship between these charter schools’ impacts on students’ 
middle school achievement and educational attainment, the study relied on five data sources: (1) 
National Student Clearinghouse (NSC) records on college enrollment and completion, (2) the 
Integrated Postsecondary Education Data System (IPEDS), (3) students’ middle school math and 
reading test scores and other administrative data, (4) a parent survey, (5) and a principal survey. Data 
from the last three sources were collected by the original study team. The data sources are summarized 
in Table A.5. 


NSC data. Data on college outcomes were obtained for students in the study sample from the 
NSC. The NSC is an organization that collects data on enrollment and completion from participating 
colleges across the United States, which must report the following information for each student: first 
name, last name, date of birth, enrollment status, dates of attendance, whether the student graduated, 
and the date of graduation. The NSC data cover 97 percent of students enrolled in degree-granting 
institutions who are eligible to accept federal student aid (Dundar & Shapiro, 2016). As of 2016, 
participation rates were higher for some types of colleges than others: 99 percent of public colleges, 
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96 of nonprofit private colleges, and 87 percent of for-profit private colleges provided data to the 
NSC. 


Names and birth dates for students in the original study sample were used to request data on their 
college enrollment and completion from the NSC and to match the two data sources. The NSC data 
spanned August 2009 to December 2017, by which point students in the study sample would have 
been out of high school for three to seven years if they had graduated on a typical schedule (Table 
A.3). 


Table A.3. Projected year of high school graduation and number of years of potential college enrollment 


Projected year of Potential years of Number of Number of 

ATlamxeuleye)| fore) |(=y2X=1 students students Total number of Percentage of 

graduation enrollment (Cohort 1) (0) ste) na") students sample 

2011 7 156 - 156 5% 

2012 6 839 237 1,076 37% 

2013 5 354 947 1,301 45% 

2014 4 O 342 342 12% 

2015 3 - 20 20 1% 
Source: Data from Gleason et al. (2010). 
Note: At the time of data collection, the most recent available college graduation data were from spring and summer 2017. 


The most recent available enrollment information was from fall of the 2017-2018 school year. 


Because of the nature of the NSC data, some missing observations are expected. In particular, 
the NSC provides data on college outcomes only for students enrolled in participating colleges, so 
students in the study sample who did not attend college were absent from the NSC data. However, 
we cannot assume that all students missing from the data set did not attend college. A student could 
be missing from the NSC data set for one of four reasons: 


1. The student did not attend college. 


2. The student’s name was incorrect in the original data set (or in the NSC data) or had changed, 
and there was no match. 


3. The student opted out of sharing his or her data under the Family Educational Rights and Privacy 
Act (FERPA). This law permits students to request that their educational data not be shared. The 
NSC reports the number of students in the returned data set who are missing because they opted 
out of sharing their records, but it does not identify these students. 


4. The student attended a college that does not provide data to the NSC (see above). 


It was not possible to determine with certainty whether students were missing from the NSC data 
because they did not attend college, which relates directly to the outcomes of interest, or for one of 
the other three reasons. Because the NSC data excluded some students who did in fact enroll in college 
(false negative matches), it is likely that the percentage of students who completed college was 
underestimated. It is also possible that a study student who did not attend college was incorrectly 
matched in the NSC data to a student with the same name and birth date (false positive matches), 
although Dynarski et al. (2015) indicate that in the NSC data, false negatives are much more common 
than false positives. 


Even if the occurrence of false negatives in the NSC data led to an underestimation of educational 
attainment in the sample, it did not necessarily lead to bias in the estimates of how charter middle 
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schools affected college outcomes. The key issue is whether the rate of occurrence of false negatives 
or false positives differed among the students admitted to charter middle schools (the lottery winners) 
versus those not admitted (the lottery losers). If these rates were the same for the two groups, the 
estimates would likely be unbiased. Several analyses were conducted to assess the potential influence 
that false positive and false negative matches could have on the estimates. 


First, the study examined whether there might be systematic differences in student names 
between lottery winners and losers that might lead those in one group to be more likely to fail to 
match. In cases where a student had two last names or a nickname in the original data file, NSC data 
were requested for the same student under multiple name variations. Although it is impossible to 
directly assess whether the lottery winners and losers had the same rate of name errors, it is possible 
to assess whether the rate of multiple name requests was the same. There was no statistically 
significant difference between the two groups (Table A.4). In addition, these data were collected 
primarily from parents before the lottery each year, and therefore any errors in this information should 
have been evenly distributed between the lottery winners and lottery losers. 


Next, the study assessed whether the proportion of students who opted not to share their data 
under FERPA differed between lottery winners and losers. The NSC indicated how many total student 
records in each group were found in the NSC database (in other words, attended college), but were 
withheld due to students opting out on the basis of FERPA. Three percent of students in both the 
lottery winner and lottery loser group files withheld their data, suggesting this was not a source of 
bias (Table A.4). 


Finally, among the students in the returned NSC data set, the study examined the distribution of 
types of colleges that students from both groups attended. If one group of students was more likely 
than the other to attend a college that does not report data to the NSC, the estimates could be biased. 
For example, if lottery losers were more likely to attend private for-profit colleges, the rate of college 
enrollment and completion among the lottery loser group might be underreported because private for- 
profit colleges are less likely to report data to the NSC than other institutions. However, the two 
groups attended public colleges (55 percent), private nonprofit colleges (13 percent), and private for- 
profit colleges (1 percent) at the same rates (Table A.4). Because for-profit colleges have a lower 
coverage rate (87 percent) than other types of colleges (96—99 percent) in the NSC data, there could 
be a larger difference between the lottery winners and lottery losers than the data suggest. However, 
given that only 1 percent of the lottery winners and losers in the NSC data attended for-profit colleges, 
it is highly unlikely that either lottery winners’ or lottery losers’ attendance at the 13 percent of for- 
profit colleges that are not included in the NSC would lead to any change in the study conclusions. 


Based on these analyses, there are no reasons to believe that the rates of false positives or false 
negatives differed systematically between the two groups. Therefore, the study team proceeded with 
the analyses without any adjustments to the NSC data. 
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Table A.4. Summary of postsecondary data requested and returned, by treatment group 


3) 24) 1) Cc) 


Mean, full Mean, lottery Mean, lottery Dy ini=asyaner=) 
Outcome sample winners losers 


Institution type where currently or most recently enrolled 


(full sample) 
Public 0.55 0.55 0.54 0.01 
Private nonprofit 0.13 0.13 0.14 -0.01 
Private for-profit 0.01 0.01 0.01 0.00 
Opted out of sharing data with the NSC@ 0.03 0.03 0.03 -0.01 
Name variations 
Multiple names requested (full sample) 0.06 0.07 0.06 0.01 
Single name requested (sample of students ever 0.94 0.93 0.93 0.00 
enrolled in postsecondary institution) 
Multiple names requested for student ever enrolled in 0.06 0.07 0.07 0.00 
postsecondary institution 
Number of students 2,873 1,723 1,150 
Note: Means are estimated at the site level and then averaged across all 31 sites in the study. The individual site-level 
estimates are weighted according to each student’s probability of assignment to the lottery winner or lottery loser 
group. 


The estimates are not statistically significant at the .05 level, two-tailed test. 


aThe National Student Clearinghouse (NSC) reports a count of the number of students in the data request file who were found in 
NSC records but had opted out of sharing their data. The means presented in this table represent the raw proportion of the lottery 
winner or lottery loser group students in the data request who opted out and are not adjusted for selection probability or site. 
Because the data requests included some of the same students multiple times with different name variations, the number of 
students in these calculations is 3,177 (1,908 in the lottery winner group and 1,269 in the lottery loser group). All other means 
in this table are weighted according to selection probability and include 2,873 students (full analytic sample) or 2,020 students 
(those who ever enrolled in any postsecondary institution). 


IPEDS data. IPEDS, a data center within the U.S. Department of Education’s National Center 
for Education Statistics, includes information on U.S. colleges, universities, and technical and 
vocational institutions. All postsecondary institutions that receive federal financial aid are required to 
complete the IPEDS survey. The IPEDS data were used for information on the types of institutions 
attended by students in the study sample (public, private nonprofit, private for-profit) and on the 
selectivity of institutions. In the few cases when the type of institution was unavailable in the IPEDS 
data, the study team manually looked up that information on the college websites and coded the 
variables. 


Student administrative records. The original study team collected data on reading and math 
test scores, attendance, and disciplinary incidents. They collected these records for the baseline (pre- 
lottery) years when students applied for charter school admission (2004—2005 for the study’s first 
cohort and 2005-2006 for the second cohort), as well as during the study’s first and second follow- 
up years for each cohort. The models that estimate the impacts of charter middle school admission 
and attendance account for students’ baseline (pre-lottery) test scores, attendance, and suspensions. 
Test score data from the second follow-up years were used to estimate the study charter schools’ 
impacts on student achievement, as well as to examine the relationship between impacts on 
achievement and impacts on students’ college enrollment and completion. 


Parent survey. Parents completed a survey when they applied for their children’s admission to 
a study charter school in spring 2005 or spring 2006. The study used data from these surveys to 
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account for family demographic and socioeconomic information of students in the models that 
compared the college enrollment and completion outcomes of lottery winners and lottery losers. 


Table A.5. Study data sources, measures, and sample 


Number 
(percentage) Number 
Number of lottery (percentage) of 
(percentage) winners with lottery losers 
Data source Key measures with valid data _—valid data with valid data 
Student data 
National Student Ever enrolled in any postsecondary 2,873 1,723 1,150 Sample for graduation 
Clearinghouse (NSC) institution (99.7) (99.8) (99.4) with bachelor’s degree 
Earned a degree (bachelor’s, outcomes was restricted 
associate’s, or certificate) or still to students who were 
enrolled as of December 2017 projected to graduate 
, from high school in 2013 
Ever enrolled in two-year college versus or earlier (n = 2,511) 
four-year college 
Number of years attended college 
during study period 
Graduated with associate’s versus 
bachelor’s degree 
Enrolled in college classes while still 
enrolled in high school 
Integrated Postsecondary Selectivity of college attended 2,857 1,712 1,145 
Education Data System (99.1) (99.2) (99.0) 
(IPEDS) 
IPEDS and NSC Whether they attended a public, private 2,873 1,723 1,150 When data were 
nonprofit, or private for-profit college (99.7) (99.8) (99.4) missing, information 
from institution websites 
was used to determine 
institution type. 
Student administrative Student scores on state assessments in 1,926 1,174 752 
records? math and reading (88.1) (90.5) (84.6) 
Parent baseline survey Family economic and socioeconomic 2,699 1,632 1,067 
characteristics (93.7) (94.6) (92.3) 
Number of students 2,882 1,726 1,156 
Charter school data 
Principal survey and Percentage of students eligible for free 31 
publicly available data or reduced-price meals (100) 
on school 
characteristics Race/ethnicity of student population 31 
(100) 
Mean baseline achievement of students 30 
(96.8) 
Common Core of Data Urbanicity of charter school 31 
(100) 
Number of charter school 31 
sites 


aThe percentages of valid second-year follow-up test scores are calculated out of a total of 2,187 (1,298 lottery winners and 889 lottery losers), 
the number of students in the 28 sites included in the impact estimates that had valid baseline (pre-lottery schools). These estimates were 
used for the correlational analyses. 
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Data on middle school characteristics. The original study team administered surveys to 
principals of the schools attended by students in the study in fall 2006 and fall 2007. The team then 
supplemented the principal survey data with data from the National Center for Education Statistics’ 
Common Core of Data and Private School Survey, the SchoolDataDirect website, and school-specific 
report cards from states’ department of education websites. These data were used to define the 
characteristics of the different charter middle schools in the study and to examine whether charter 
schools with different characteristics (for example, schools that serve higher versus lower proportions 
of disadvantaged students) affected students’ college enrollment and completion differently. 


Analysis sample 


For the main analysis of impacts of the study’s charter middle schools on college outcomes, the 
sample included 2,873 students in 31 sites (which together included 35 charter schools). A total of 
2,904 students in 32 sites (36 charter schools) participated in the original study. Information necessary 
for matching to the NSC data (names and dates of birth) were missing for nine students, so these 
students were excluded from the NSC data request and the analyses. In addition, one charter school 
site (with 22 students) was also dropped because there were too few students in the lottery loser group 
for a reliable analysis. Of the 2,873 students included in the analysis, 1,723 were in the lottery winner 
group and 1,150 were in the lottery loser group (see Figure A.1). 


For the correlational analysis that compared impacts on student achievement to impacts on 
college outcomes, the same analysis sample was used as in the original study, to the extent possible. 
The students without valid baseline (pre-lottery) test scores and without second-year follow-up test 
scores were omitted from the analysis. Four charter middle school sites were also dropped because 
there were fewer than five students in the lottery winner or lottery loser groups, which was consistent 
with the approach used in Gleason et al. (2010). The sample of students who had valid pre-lottery and 
second-year follow-up scores and met the other requirements included 1,913 for reading, 1,908 
students for math, and 1,919 for either math or reading. For the main achievement measure, the study 
team averaged reading and math scores if a student had both, used the reading score alone if the 
student was missing the math score, and used the math score alone if the reading score was missing. 
Because more students had reading than math scores, the sample of 1,913 students was used to 
estimate the college impacts for the correlations. Of the 1,913 students included in this analysis, 1,169 
were lottery winners and 744 were lottery losers (see Figure A.1). 
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Outcome measures 


To learn about the effects of the study’s charter middle schools on students’ college enrollment 
and completion, the study examined impacts on two primary outcomes: 


Ever enrolled in any college. Student appeared in the NSC data set at any point in time 
between August 2009 and December 2017. 


Obtained degree or currently enrolled. Student completed college (earned a bachelor’s 
degree, associate’s degree, or certificate) or was enrolled as of December 2017. Any student 
who had dropped out of college was coded as not having obtained a degree or not currently 
enrolled. 


An exploratory analysis was also conducted to help better understand the impacts of the two main 
outcomes. 


The exploratory outcomes related to college enrollment include: 


Ever enrolled in two-year college. Student enrolled in a two-year college at any point in time 
between August 2009 and December 2017. 


Ever enrolled in four-year college. Student enrolled in a four-year college at any point in 
time between August 2009 and December 2017. 


Selectivity of college. Colleges were classified according to their reported acceptance rate in 
IPEDS: highly selective (1-25 percent acceptance rate), selective (26—50 percent acceptance 
rate), somewhat selective (51—75 percent acceptance rate), or less selective (76—100 percent 
acceptance rate). A student who enrolled in more than one college appears in only one 
category, reflecting the most selective college he or she attended. 


Enrolled in college courses during high school. Student enrolled part-time in college classes 
during the school year, within the two years prior to his or her projected high school graduation 
date. 


College type where currently or most recently enrolled. The NSC data classified colleges 
as public or private, and the IPEDS data provided information on whether the private colleges 
were nonprofit or for-profit. When the institution type was missing from the IPEDS data, the 
study team looked at the institution websites and manually coded these cases. 


The exploratory outcomes related to college completion include: 


Number of years completed in any college. Number of academic years (or fraction thereof) 
that student completed at any college from August 2009 through December 2017. Following 
U.S. government guidelines, one full academic year was defined as 28 weeks of full-time 
enrollment, 56 weeks of half-time enrollment, or 112 weeks of enrollment less than half time 
(Dynarski, Hemelt, & Hyman, 2015). If a student was enrolled concurrently in more than one 
college, full-time enrollment was recorded, but not more. (In other words, enrollment was 
capped at 100 percent at any given time.) Ifa student withdrew from college during a semester, 
that enrollment period was not counted. 


Number of years completed at two-year college. Number of academic years (or fraction 
thereof) that student completed at a two-year college from August 2009 through December 
2017. 
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e Number of years completed at four-year college. Number of academic years (or fraction 
thereof) that student completed at a four-year college from August 2009 through December 
2017, prior to obtaining a bachelor’s degree. (In other words, time spent working toward an 
advanced degree was not counted.) 


e Graduated with associate’s degree (two-year college). Student completed degree at a two- 
year college. If the student also completed a bachelor’s degree, only the highest degree 
(bachelor’s) was counted. 


e Graduated with bachelor’s degree (four-year college). Student completed degree at a four- 
year college. If the student also completed an associate’s degree, only the highest degree 
(bachelor’s) was counted. 


e Graduated with bachelor’s degree within four years of projected high school graduation. 
Student completed degree at a four-year college within four years. The sample for this variable 
is limited to the 87 percent of students who were projected to graduate from high school in 
2013 or earlier. 


Statistical power 


To assess the statistical power of the design, we examined the ability of the analysis to detect 
impacts of admission to a study charter school on college enrollment and completion, the key 
outcomes of interest. To the extent possible, these calculations used the actual sample sizes and design 
parameters that were used in the original study, based on the objective of that study to provide 
estimates of the impact of charter middle school admission on student achievement (see Appendix D 
of Gleason et al., 2010). Because this study is a follow-up, the sample sizes could not be adjusted 
based on the power calculations. In calculating the minimum detectable effects, we used an 80 percent 
power level and a 5 percent level of statistical significance. This study has the power to detect 
moderate to large impacts of charter middle school admission on college enrollment and completion. 
The minimum detectable effect is 0.06 (or 6 percentage points) for impacts on college enrollment, 
which assumes a control group proportion of 0.69, the proportion of students who completed high 
school and immediately enrolled in college in 2015 (McFarland et al., 2018). For impacts on college 
completion, the minimum detectable effect is 0.07 (or 7 percentage points), which assumes a control 
group proportion of 0.46, the proportion of 25- to 29-year-olds with an associate’s degree or higher 
in 2017 (McFarland et al., 2018). Thus, the study did not have the statistical power to reliably detect 
small impacts of being admitted to a charter middle school in the study on college enrollment or 
completion—impacts less than 6 percentage points for the former and 7 percentage points for the 
latter. 


Analytic approach 


This section describes the primary analyses that were conducted to answer the study research 
questions. These include the main impact and student subgroup analyses, the school subgroup 
analyses, and the correlational analyses. Many of these analyses used the same approach and models 
as the original IES study of charter middle schools (Gleason et al., 2010). This section provides the 
key information about the analytic approach, but additional details are provided in that report. 


Estimating the impact of being admitted to a charter middle school (Finding 1) 


The first step of the main impact analysis was to estimate the impact of the study’s charter middle 
schools on students’ college enrollment and completion for the full study sample. For consistency 
with the approach used in the original study (Gleason et al., 2010), a two-stage process was used to 
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estimate the average impact of the charter middle schools. The study team first estimated impacts in 
each of the 31 charter school sites included in the analysis and then calculated the average impact 
across these sites. 


Estimating site-level impacts. To examine the effect of winning a charter school lottery, 
regardless of whether the student actually attended that school, the study used a regression model to 
compare the average (mean) outcome among lottery winners with the average (mean) outcome among 
lottery losers. The model adjusted for the characteristics of students at the time of the lottery. 


The basic form of the model is 
Cl) yj=a)+X,B+6,T, +8, 


where y, is the outcome of interest for student i in charter school site j, a, is a separate intercept term 
for each charter school site, x, includes a set of characteristics of student i in site j, 7, represents the 


student’s treatment status (lottery winner or lottery loser) and indicates whether or not student 7 was 
admitted to a charter school in site j, ¢, is arandom error term that reflects the influence of unobserved 


factors on the outcome, and # and 6, are parameters or set of parameters to be estimated. The 
estimated coefficient 5, represents the impact of being admitted to a charter middle school in site /. 


Sample weights, created by the original study team, were used in the impact estimation to account 
for a variety of factors that influenced the probability that a student would be admitted to a charter 
middle school. These factors include the number of open seats at the school, the number of students 
offered admission based on the initial lottery as well as from the waiting list after the lottery, whether 
parents gave consent for their children to participate in the study before the lottery, whether siblings 
applied together and each school’s policy toward sibling acceptance, whether schools stratified their 
lotteries on the basis of predetermined student characteristics (that is, whether they conducted random 
assignment separately for different groups of students such as students from particular feeder 
schools), and whether students applied to more than one charter middle school in the study. 


The impacts were estimated using linear regression models for both continuous and binary 
outcomes. Most of the postsecondary outcomes in this study are binary — that is, they have two 
possible values, such as whether or not a student attended a four-year college or obtained a bachelor’s 
degree. In these models, the error variance is not constant, so robust standard errors were used to 
correct for heteroscedasticity. Several sensitivity tests were also conducted using different model 
specifications described in Appendix C. 


The models controlled for student background characteristics and baseline (pre-lottery) test 
scores (Table A.2). This meant that variables for which there were statistically significant differences 
between lottery winners and lottery losers were controlled for, as suggested by guidance from the 
U.S. Department of Education’s What Works Clearinghouse. Also, these are the same variables used 
in the original IES study, providing consistency across the two studies. The models in both studies 
also controlled for students’ age at the start of school, whether students were young or old for their 
grade, and their cohort. These variables accounted for the number of years of possible college 
enrollment and for the fact that a student’s college outcomes may have varied with the number of 
years that had passed since the student graduated from high school (Table A.3). 


Accounting for missing data. Some baseline (pre-lottery) data were missing for some students 
in the sample. To account for missing baseline data, the sample median or mode was imputed for each 
baseline variable. In other words, the missing value was replaced with a valid value—the median or 
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mode value of that variable among all students in the analysis sample with valid data. In the impact 
models, the study team used the imputed baseline variables and included a missing dummy variable 
for each baseline variable with imputed values. 


Averaging impacts across sites. After estimating the impact of each charter middle school, the 
impact estimates were averaged across sites to estimate the overall impact of the study’s charter 
middle schools on each outcome. The average overall impact 5 was calculated as the sum of the 
impact of each site 6, divided by the number of sites J: 


n [4a 
(2) 6=-y6, 
mpag, 


In this calculation, each site was given equal weight, so each charter school in the study contributed 
equally to the overall impact estimate. As a sensitivity test, the average was re-calculated, weighting 
each site according to the precision of its impact estimate. See Appendix C for a more detailed 
description. 


Estimating the impact of attending a charter middle school. The impact of actually attending 
the charter middle schools in the study, rather than the impact of simply being offered admission, was 
also estimated. This is known as a treatment-on-the-treated estimate. To do this, the treatment status 
indicator—whether a student was a lottery winner or lottery loser—was used as an instrumental 
variable for charter middle school attendance.”? As described in Gleason et al. (2010), using this 
instrumental variable can provide an unbiased estimate of the effect of attending a charter middle 
school when (1) winning the charter lottery is highly predictive of whether a student actually attends 
a charter school and (2) study charter schools were equally effective for lottery winners and lottery 
losers who attended them. 


To obtain the treatment-on-the-treated estimate, we estimated a two-stage least squares model. 
First, a variable indicating whether each student attended a study charter middle school was regressed 
on the treatment status indicator and the other variables. From this regression model, each student’s 
predicted charter school attendance was calculated. Second, the outcome variable was regressed on 
predicted charter school attendance and the other variables in the model. The coefficient on predicted 
charter school attendance in that model represents the impact of attending a study charter middle 
school. 


Estimating the impact for subgroups of students. The study also examined whether the charter 
middle schools in the study had different effects on different types of students. The subgroup analyses 
focus on the same student subgroups as the original study to be able to determine (a) whether students 
who have traditionally experienced lower rates of college attendance and completion fare better if 
they are offered admission to charter schools than if they attend traditional public schools and (b) 
whether the subgroup findings for postsecondary outcomes were consistent with those that the 
original study found for student achievement. 


Student groups were defined based on: 
e Students’ eligibility for free or reduced-price school meals 
e Race/ethnicity 


e Baseline (pre-lottery) math and reading achievement above or below the sample median. 
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To estimate the effects on student subgroups, the main regression model was used, with one 
variation: 


(3) yi=HUtXiP'+OT, +7jS+¢',Ti,St+e'i, 


where S is an indicator for whether the student is in the subgroup of interest, and the other variables 
are as defined as in Equation (1). The study team calculated the impacts for each subgroup separately 
and then tested the significance of the difference between the two groups. 


Estimating the impact for groups of schools. The study also examined whether some types of 
charters were more effective than others. It is important for policymakers to understand the 
effectiveness of charter schools that serve different types of students and those in urban versus 
nonurban locations, given that research suggests the impacts on achievement can vary along with 
these characteristics. The models and approach for estimating the effects of different groups of 
charters were the same as the student subgroup model described above. Charter school subgroups 
were defined based on the following charter school characteristics: 


e Proportion of students who were eligible for free or reduced-priced school meals (above or 
below the sample median) 


e Proportion of students who were white or non-white (above or below the median) 
e Urbanicity of the school (urban versus suburban or rural) 


e Average baseline (pre-lottery) reading and math scores (above or below the median). 


In addition to the possibility that the subgroups of charter middle schools defined by the above 
characteristics had different impacts on college outcomes than the average charter middle school in 
the study, there could be differences in charter schools’ impacts based on other characteristics. To 
investigate the more general possibility that different charter schools have different impacts on 
college outcomes, the study estimated variation in impacts across the study schools. Specifically, the 
study tested whether impacts on each of the college outcomes were the same across all study charter 
schools, or whether different study charter schools had different impacts. 


The estimates of the charter middle schools’ impacts varied from -0.25 to 0.16 for college 
enrollment and -0.20 to 0.42 for completion, but one factor leading to this variation was sampling 
error. The impacts were based on the samples (number of students who went through lotteries) in 
each charter school site, and in some cases, the samples were small. As a result, some of the site-level 
estimates were not very precise and differences in the impacts of any two charter schools could have 
resulted from this sampling error. To determine whether there was a statistically significant difference 
in impact estimates across sites, a Q-test for homogeneity across all sites was conducted (Gleason et 
al., 2010). The study team was unable to reject the null hypothesis of no differences in impacts across 
schools at the 0.05 level for either the college enrollment impacts (p-value = 0.412) or the college 
completion impacts (p-value = 0.145). This means that there were not statistically significant 
differences in college impacts across the charter middle schools for either of the primary college 
outcomes. See Appendix C for details. 
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Estimating the relationship between impacts on student achievement and college 
enrollment and completion (Finding 2) 


Finally, the study explored whether the charter middle schools that had positive impacts on 
students’ academic achievement also had positive long-term impacts on students’ college enrollment 
and completion, or whether short-term impacts on test scores were unrelated or negatively related to 
long-term impacts. 


To examine the relationship between middle school achievement and college outcomes for 
individual students, the study estimated simple correlations between student middle school 
achievement and three postsecondary outcomes: (1) ever enrolled in college, (2) obtained a degree or 
currently enrolled, or (3) ever attended a four-year college. The correlations are presented in two 
ways. Middle school achievement was measured in three different ways—math scores, reading 
scores, and the average of math and reading scores, resulting in three separate analyses. Each 
postsecondary outcome was regressed on students’ second-year follow-up (Year 2) math and/or 
reading test scores, using ordinary least squares and robust standard errors. As an alternative way of 
examining this relationship, simple Pearson correlation coefficients were estimated. 


Next, the study estimated the correlations between the impacts of admission to study charter 
schools on students’ math and reading achievement and the impacts of these schools on the college 
outcomes. The approach followed a similar correlational analysis presented in Angrist et al. (2016). 
It involved separately estimating impacts on six outcomes for students who applied to each charter 
middle school in the original study (Year 2 middle school math scores, Year 2 middle school reading 
scores, combined Year 2 math and reading scores, ever enrolled in college, obtained a degree or 
currently enrolled, and ever enrolled in four-year college). 


Once impacts were estimated for each of the six outcome variables for each charter school lottery, 
the study team examined how impacts on students’ math and reading achievement related to each 
postsecondary outcome. The estimated college enrollment impact was regressed on the estimated 
impacts on students’ Year 2 math and reading test scores, using ordinary least squares. The estimated 
college completion impact was also regressed on these same achievement impact estimates. The study 
reports the coefficient estimates from this regression, which represent the predicted percentage point 
change in a college outcome, given a one-unit increase in achievement impacts.”* 


In addition, the Pearson correlation between the estimated impact on reading and math 
achievement and the impacts on each of the postsecondary outcomes were computed and reported. 
The correlation coefficient describes the strength of the relationship between test score impacts and 
college outcome impacts, accounting for the fact that more than one test score impact measure was 
included in the model. A correlation of 1 would indicate that impacts on middle school achievement 
perfectly predict impacts on college outcomes; a correlation of 0 would indicate that there is no 
relationship between the middle school impacts and college impacts. 
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APPENDIX B: SUPPORTING DETAIL FOR KEY FINDINGS 


This appendix presents detailed results supporting the main findings described in the evaluation 
brief. 


Supporting Detail for Overall Impacts on College Enrollment and Completion 
(Finding 1) 


The first set of tables (Tables B.1 through B.3) show the full set of estimates of how admission 
to and enrollment in study charter middle schools affected college outcomes. Table B.1 presents 
evidence on how both being admitted to and attending a study charter school affected the main 
outcomes of college enrollment and completion as well as each of the exploratory college outcomes, 
among the sample of 2,873 students in 31 charter school sites. The exploratory outcomes provide 
additional information about the study’s primary outcomes and allow us to better understand the 
charter middle schools’ effects (or lack of effects) on the primary outcomes. For reference, the table 
also includes estimates of how study charter schools affected students’ middle school achievement. 
The study also examined how admission to these study charter schools affected each of these 
outcomes for key subgroups of students, including students who had different background 
characteristics (Table B.2) and achievement levels prior to charter school admission (Table B.3). 
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Table B.1. Being admitted to or attending a study charter school did not affect students’ college outcomes 


Tan) o=\e1 me) ol=)iay-a-1elaayinc-le mob VaveM-|an-\ale|lat-a-M-1a0 (eh meat-lac-yamany(o(e|(=m=xe) alee) Mola mexe)|(-v:<-M-1alco)| luat-aiar-]ale| 
completion outcomes 


Impact of admission offer Tig) ez \e1mey m= 14-1 alel-]aer=) 


(1) (2) if) (3) 
Mean, lottery Mean, lottery Difference Adjusted impact 
Outcome winners losers (impact estimate) estimate 


Primary outcomes 

Ever enrolled in any postsecondary institution 0.69 0.69 0.00 0.923 -0.03 0.733 

Obtained a certificate, associate’s degree, or bachelor’s 0.48 0.47 0.014 0.634 0.02 0.696 
degree or currently enrolled 

Exploratory outcomes related to college enrollment 

Ever enrolled in postsecondary institution 

2-year college 0.32 0.35 -0.03 0.188 -0.03 0.649 

4-year college 0.55 0.52 0.02 0.296 0.04 0.409 

Type of institution where currently or most recently 

enrolled 

Public 0.55 0.54 0.00 0.918 -0.01 0.885 

Private nonprofit 0.13 0.14 -0.01 0.729 -0.03 0.637 

Private for-profit 0.01 0.04 0.00 0.945 0.04 0.485 

Selectivity of postsecondary institution 

Highly selective (1-25% acceptance rate) 0.03 0.02 0.04 0.130 0.03 0.151 

Selective (26-50% acceptance rate) 0.13 0.13 0.00 0.995 0.014 0.768 

Somewhat selective (51-75% acceptance rate) 0.25 0.23 0.02 0.330 0.014 0.845 

Less selective (76-100% acceptance rate) 0.27 0.30 -0.03 0.133 -0.08 0.320 

Dual enrollment participation 0.09 0.10 -0.01 0.543 -0.05 0.635 

Exploratory outcomes related to college completion 

Graduated from postsecondary institution@ 

Associate’s degree (2-year college) 0.03 0.05 -0.01 0.209 -0.05 0.575 

Bachelor’s degree (4-year college) 0.27 0.29 -0.02 0.361 -0.04 0.288 

Graduated with bachelor’s degree within 4 years of 0.24 0.25 -0.01 0.731 -0.02 0.596 
projected high school graduation 
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Tan) oy=\e1me)im ol=)iay-a-1elaninc-lemcob-Vavem-|nn-\ale|lat-a-M-1a0 (eh meat-lac-ymany(o(e|(=m=xe) alee) Mola mexe)|(-v:<-M-1a]ce)|luat-almr-]a1° | 
Core) pay e)(=1tle)ameleinere) pales) 


Impact of admission offer Tia) ez \e1mey m= 14-1 ale[-]aler=) 


ce) 7) ‘€)) (4) (5) (3) 
Mean, lottery Mean, lottery Difference p-value Adjusted impact p-value 
Outcome winners losers (impact estimate) estimate 


Graduated or currently enrolled 


Graduated with associate’s degree or currently enrolled in 0.07 0.06 0.014 0.204 0.06 0.411 
2-year college 
Graduated with bachelor’s degree or currently enrolled in 0.41 0.40 0.014 0.805 -0.02 0.815 


4-year college 


Number of years completed 


At 2-year college 0.34 0.38 -0.04 0.423 -0.02 0.837 
At 4-year college 1.89 1.81 0.08 0.412 0.06 0.715 
At any postsecondary institution 2.36 2.34 0.02 0.847 0.02 0.938 
Middle school Year 2 achievement’ 

Reading 0.31 0.38 -0.07 0.032+ -0.08 0.117 
Math 0.32 0.38 -0.06 0.136 -0.08 0.202 
Number of students 1,723 1,150 2,873 2,822 

Number of sites 31 31 


Note: In both the lottery winner and lottery loser groups, 3 percent of students who attended a postsecondary institution opted out of sharing their data. In the student-level 
postsecondary data file, these students are indistinguishable from the students who never enrolled in a postsecondary institution, and are counted as if they never 
enrolled. Therefore, estimates presented in this table underestimate overall enrollment by approximately 3 percentage points. 


We estimated means at the site level and then averaged across all 31 sites in the study, with the averages weighted according to each student’s probability of 
assignment to the lottery winner or lottery loser group. 


The p-value represents the probability of getting the estimated difference or impact if, in fact, the true difference or impact was equal to O. 


The results are presented in proportions, with the exception of the number of years completed outcomes, which are presented in years, and middle school Year 2 
achievement outcomes, which are presented in standard deviations. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled immediately after high school and for whom we have 
at least four years of additional data to measure college completion (n = 2,511 for the impact of admission offer estimates and n = 2,466 for the impact of attendance estimates). 


>The achievement impact estimates are from Table IV.1 of Gleason et al. (2010). The middle school Year 2 achievement sample is 1,920 students. 
*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
+Difference significantly different from zero at the .05 level, two-tailed test. 

++Difference significantly different from zero at the .01 level, two-tailed test. 
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Table B.2. Being admitted to a study charter school did not affect students’ college outcomes differently for students with different 
characteristics 


Tag) oys(oumey im ol-\ay-ar=Le[aaliaccvomcon-m-1uU(e\mear-an-amnaliece|(= eye slele)me)amere)| (-1X-M-Valce)|lnal- 118 
FV alo mere) aaye)(=1u(o) amelU| core) a al=\-yam o)yar-) au e(-1alm er-(e1.¢-4c0l0 ale Mesat-le-[eik-) ati e 


Certification for free or reduced-price meals 


(1) 7) 1) (4) (5) (2) 
Not certified Certified Difference White, non- Nonwhite Difference 
Hispanic and/or 
Outcome Hispanic 


Primary outcomes 


Ever enrolled in any postsecondary institution -0.02 0.01 0.03 -0.02 0.014 0.03 
Obtained a certificate, associate’s degree, or bachelor’s degree or currently -0.01 -0.01 0.00 -0.04 0.00 0.04 
enrolled 


Exploratory outcomes related to college enrollment 


Ever enrolled in postsecondary institution 


2-year college -0.02 -0.03 -0.01 -0.04 0.01 0.05 
4-year college 0.00 0.02 0.02 0.01 0.00 -0.01 
Type of institution where currently or most recently enrolled 

Public 0.00 0.02 0.02 0.03 0.01 -0.02 
Private nonprofit -0.02 -0.01 0.014 -0.04~ -0.01 0.04 
Private for-profit 0.00 0.00 0.00 0.00 0.00 0.00 
Selectivity of postsecondary institution 

Highly selective (1-25% acceptance rate) 0.02~ 0.00 -0.02 0.00 0.03 0.03 
Selective (26-50% acceptance rate) -0.01 0.01 0.01 0.02 0.00 -0.02 
Somewhat selective (51-75% acceptance rate) 0.01 0.06~ 0.05 0.01 0.01 0.00 
Less selective (76-100% acceptance rate) -0.04 -0.05 -0.01 -0.05~ -0.03 0.02 
Dual enrollment participation -0.02 0.03 0.05 0.00 0.00 0.00 


Exploratory outcomes related to college completion 


Graduated from postsecondary institution@ 


Associate’s degree (2-year college) -0.01 -0.03 -0.02 -0.02 0.00 0.02 

Bachelor’s degree (4-year college) -0.04 0.00 0.05 -0.02 -0.06~ -0.04 

Graduated with bachelor’s degree within 4 years of projected high school -0.02 -0.01 0.01 0.00 -0.04 -0.04 
graduation 

Graduated or currently enrolled 

Graduated with associate’s degree or currently enrolled in 2-year college 0.03~ -0.01 -0.04~ -0.01 0.04~ 0.05+ 

Graduated with bachelor’s degree or currently enrolled in 4-year college -0.03 0.02 0.06 -0.03 -0.02 0.01 
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Certification for free or reduced-price meals 


3) 24) 1) (4) t)) (3) 


Not certified Certified Difference White, non- Nonwhite Difference 
Hispanic and/or 
Outcome Hispanic 
Number of years completed 
At 2-year college 0.014 -0.07 -0.08 -0.06 0.03 0.09 
At 4-year college -0.06 0.11 0.17 0.02 -0.13 -0.15 
At any postsecondary institution -0.07 0.06 0.13 -0.08 -0.114 -0.03 
Middle school Year 2 achievement 
Reading -0.12** 0.05 0.17+ -0.07 -0.08 -0.02 
Math -0.14* 0.17* 0.31++ -0.09 -0.03 0.05 
Number of students 2,005 819 2,824 1,587 1,218 2,805 
Number of sites 31 19 25 25 
Note: The results are presented in proportions, with the exception of the number of years completed outcomes, which are presented in years, and middle school Year 2 


achievement outcomes, which are presented in standard deviations. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled immediately after high school and for whom we have 
at least four years of additional data to measure college completion (n = 2,511 for the impact of admission offer estimates and n = 2,466 for the impact of attendance estimates). 


*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 

+Difference significantly different from zero at the .05 level, two-tailed test. 
++Difference significantly different from zero at the .01 level, two-tailed test. 

~Estimate or difference significantly different from zero at the .10 level, two-tailed test. 
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Table B.3. Being admitted to a study charter school did not affect students’ college outcomes differently for students with different pre-charter 
school achievement levels 


Tan) oys(oumeoy im of-\ay-ar=\elsaliac-vomcon-M-1u0(o\mear-lan-yamanliecel(= mye alele)me)amere)| (-1X-M-\alce)| lnal-vaime= lace mexedaay e)(sia(e)al 
outcomes, by student baseline achievement 


Reading achievement Math achievement 


(1) (73) ()) (4) (5) ((3)) 
Outcome 15X=1 (oa natsxe [t-1a) Above median Difference 13Y=1 (oa natsxe [t-1a) Above median Difference 


Primary outcomes 


Ever enrolled in any postsecondary institution -0.04 0.04 0.07~ -0.01 0.04 0.05 
Obtained a certificate, associate’s degree, or -0.01 0.03 0.04 0.01 0.00 -0.01 
bachelor’s degree or currently enrolled 


Exploratory outcomes related to college enrollment 


Ever enrolled in postsecondary institution 


2-year college -0.04 -0.04 0.00 -0.05~ -0.01 0.04 
4-year college -0.02 0.08* 0.11+ 0.01 0.06~ 0.05 
Type of institution where currently or most recently 

enrolled 

Public -0.03 0.04 0.07 -0.02 0.05 0.07 
Private nonprofit -0.01 0.01 0.014 -0.01 0.00 0.01 
Private for-profit 0.00 -0.01 -0.01 0.01 -0.01 -0.02~ 
Selectivity of postsecondary institution 

Highly selective (1-25% acceptance rate) 0.00 0.01 0.014 0.014 0.014 0.00 
Selective (26-50% acceptance rate) -0.01 0.02 0.03 0.00 0.03 0.02 
Somewhat selective (51-75% acceptance rate) -0.01 0.04 0.05 0.00 0.03 0.03 
Less selective (76-100% acceptance rate) -0.02 -0.04 -0.02 -0.02 -0.03 0.00 
Dual enrollment participation 0.014 -0.01 -0.03 0.00 0.00 0.00 


Exploratory outcomes related to college enrollment 


Graduated from postsecondary institution@ 


Associate’s degree (2-year college) -0.02 0.01 0.03 -0.02~ 0.01 0.03~ 
Bachelor’s degree (4-year college) -0.04 -0.03 0.014 -0.03 -0.01 0.02 
Graduated with bachelor’s degree within 4 years of -0.02 -0.02 0.00 -0.01 -0.01 -0.01 


projected high school graduation 
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outcomes, by student baseline achievement 


Reading achievement Math achievement 


Outcome oh Tater Hise aie Ahem i eee a sao peer 


Graduated or currently enrolled 


Graduated with associate’s degree or currently 0.02 0.01 -0.014 0.02 0.00 -0.03 
enrolled in 2-year college 
Graduated with bachelor’s degree or currently enrolled -0.02 0.03 0.05 0.00 0.01 0.01 


in 4-year college 


Number of years completed 


At 2-year college -0.02 -0.07 -0.05 -0.08 0.00 0.08 
At 4-year college -0.13 0.21 0.34~ 0.07 0.09 0.01 
At any postsecondary institution -0.19 0.15 0.34 -0.05 0.06 0.11 
Middle school Year 2 achievement 

Reading -0.02 -0.13* -0.114 0.01 -0.10 -0.12 
Math 0.05 -0.114 -0.16 0.08 -0.10 -0.18 
Number of students 1,714 1,143 2,857 1,706 1,148 2,854 
Number of sites 31 27 31 27 

Note: The results are presented in proportions, with the exception of the number of years completed outcomes, which are presented in years, and middle school Year 2 


achievement outcomes, which are presented in standard deviations. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled immediately after high school and for whom we have 
at least four years of additional data to measure college completion (n = 2,511 for the impact of admission offer estimates and n = 2,466 for the impact of attendance estimates). 


*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 

+Difference is significantly different from zero at the .05 level, two-tailed test. 
++Difference is significantly different from zero at the .01 level, two-tailed test. 

~Estimate or difference is significantly different from zero at the .10 level, two-tailed test. 
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Table B.4 shows the impact of being admitted to charter schools with different characteristics on 
students’ college outcomes. In particular, the table shows the full set of results for charter sites serving 
more versus fewer economically advantaged students, more non-white versus more white students, 
and low-achieving versus high-achieving students, as well as sites within large urban areas versus 
outside of large urban areas. Figures B.1 and B.2 show these results graphically. 
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Table B.4. Subgroups of charter middle schools with different characteristics had similar effects on college outcomes 


Tan} exs\e1me)im ol-\iat-M-Lolnalinn-vomcom-M-1t0(0\’meiar-lac-lmanl(ele|(-m-vevaele) Mela mere) i(-¥--M-1alce)|[aat-valme- lave mee )an] o)(=1t(e)amol¥inere) pal-t-em oy me at-1an-lm@cve a cele) Keval-1e-(et-) acl (e 


Percentage eligible for free or reduced- 


price meals Percentage of white students Mean pre-lottery achievement Urbanicity 
(4) (2) (3) (4) CS)] (3) (7) (8) (<)) (ene) (11) (G3) 
Sites serving Sites serving Diff- Sites Sites Diff- Sites Sites Diff- Sites Sites in Diff- 
more fewer erence serving serving erence serving serving erence | outside of large erence 
economically economically more non- rate) (=) low- high- large urban 
advantaged advantaged white white achieving achieving ldey= 1a) areas 
Outcome students students students students students students areas 
Primary outcomes 
Ever enrolled in any postsecondary -0.01 0.04 0.02 0.02 -0.03 -0.05 0.00 -0.01 0.00 0.00 -0.01 -0.01 
institution 
Obtained a certificate, associate’s 0.00 0.03 0.03 0.05 -0.04 -0.09+ 0.03 -0.02 -0.05 0.00 0.02 0.02 
degree, or bachelor’s degree or 
currently enrolled 
Exploratory outcomes related to college 
enrollment 
Ever enrolled in postsecondary 
institution 
2-year college -0.06~ 0.00 0.06 0.01 -0.07* -0.08~ -0.02 -0.05~ -0.03 -0.05~ 0.01 0.06 
4-year college 0.00 0.05 0.05 0.06~ -0.02 -0.07 0.03 0.01 -0.02 0.02 0.04 0.02 
Type of institution where currently or 
most recently enrolled 
Public 0.02 -0.02 -0.05 -0.01 0.01 0.02 -0.02 0.02 0.03 0.01 0.00 -0.01 
Private nonprofit -0.03 0.02 0.04 0.02 -0.04 -0.06~ 0.01 -0.02 -0.03 0.00 -0.01 -0.01 
Private for-profit -0.01 0.01 0.02+ 0.01 -0.01 -0.02 0.00 0.00 0.00 0.00 0.01 0.01 
Selectivity of postsecondary institution 
Highly selective 0.00 0.02** 0.01 0.02** 0.00 -0.03~ 0.01 0.01 0.00 0.01 0.02~ 0.01 
(1-25% acceptance rate) 
Selective 0.00 0.00 -0.01 0.02 -0.02 -0.04 0.00 0.00 0.00 -0.01 0.03 0.04 
(26-50% acceptance rate) 
Somewhat selective 0.01 0.03 0.02 0.01 0.03 0.01 0.04~ 0.00 -0.05 0.02 0.03 0.01 
(51-75% acceptance rate) 
Less selective -0.03 -0.03 0.00 -0.04 -0.03 0.00 -0.05 -0.01 0.04 -0.01 -0.08*  -0.07 
(76-100% acceptance rate) 
Dual enrollment participation -0.03~ 0.02 0.05~ 0.00 -0.03 -0.03 0.00 -0.01 -0.01 0.00 -0.03 -0.03 
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Percentage eligible for free or reduced- 


price meals Percentage of white students Mean pre-lottery achievement Urbanicity 


ce) (2) i) (4) )) (3) 0) (3) )) (10) (11) (12) 


Sites Diff- Sites 
serving erence serving 
rate) (=) low- 

white achieving 
students students 


Sites 
serving 
more non- 
white 
students 


Sites Diff- 
serving erence 
high- 
achieving 
students 


Sites in Diff- 
large erence 
urban 
areas 


Sites 
outside of 
large 
idey= 1a) 
areas 


Sites serving 
more 
economically 
advantaged 
students 


Sites serving Diff- 
fewer erence 

economically 

advantaged 


Outcome students 


Exploratory outcomes related to college 

completion 

Graduated from postsecondary 

institution? 

Associate’s degree (2-year college) -0.02~ -0.01 0.014 0.00 -0.03~ -0.03 0.00 -0.01 -0.01 -0.01 -0.01 0.00 

Bachelor’s degree (4-year college) -0.01 -0.03 -0.01 -0.01 -0.03 -0.02 -0.03 -0.01 0.02 -0.01 -0.03 -0.02 

Graduated with bachelor’s degree 0.01 -0.03 -0.04 -0.01 -0.01 0.00 -0.03 0.02 0.05 0.00 -0.02 -0.02 
within 4 years of projected high 
school graduation 

Graduated or currently enrolled 

Graduated with associate’s degree or 0.00 0.04~ 0.04~ 0.04* -0.01 -0.05+ 0.02 0.00 -0.02 0.01 0.02 0.01 
currently enrolled in 2-year college 

Graduated with bachelor’s degree or 0.01 0.00 -0.01 0.03 -0.03 -0.06 0.02 -0.01 -0.03 0.00 0.01 0.01 
currently enrolled in 4-year college 

Number of years completed 

At 2-year college -0.02 -0.06 -0.04 -0.02 -0.06 -0.04 0.00 -0.08 -0.08 -0.03 -0.04 -0.01 

At 4-year college 0.09 0.07 -0.02 0.14 0.00 -0.14 0.10 0.03 -0.06 0.10 0.04 -0.06 

At any postsecondary institution 0.04 0.00 -0.04 0.15 -0.14 -0.28 0.12 -0.12 -0.23 0.03 0.00 -0.02 

Middle school Year 2 achievement 

Reading -0.11 0 0.14 -0.08* -0.03 0.05 0.03 -0.15** -0.17++| -0.08* -0.02 0.06 

Math -0.24** 0.18** 0.41++ 0.01 -0.11 -0.12 0.12* -0.21** -0.33++) -0.14** 0.16* 0.30++ 

Number of students 1,366 1,507 2,873 | 1,676 1,197 2,873 1,448 1,405 2,853 1,903 970 2,873 

Number of sites 16 15 31 17 14 31 15 15 30 21 10 31 

Note: The results are presented in proportions, with the exception of the number of years completed outcomes, which are presented in years, and middle school Year 2 achievement outcomes, 


which are presented in standard deviations. 


aThe sample for the graduation outcomes was limited to students projected to have completed college if they enrolled immediately after high school (n = 2,511). 
>The achievement impact estimates are from Tables G.11, G.14, G.15, and G.16 of Gleason et al. (2010). The reported sample sizes vary depending on the charter school characteristic. 
*Significantly different from zero at the .05 level, two-tailed test. 
* Significantly different from zero at the .01 level, two-tailed test. 
+Difference is significantly different from zero at the .05 level, two-tailed test. 
++Difference is significantly different from zero at the .01 level, two-tailed test. 
~Estimate or difference is significantly different from zero at the .10 level, two-tailed test. 
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Figure B.1. Subgroups of charter schools with different characteristics had similar effects on college 
enrollment: Percentage of lottery winners and losers who enrolled in college, by charter school 
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Note: Subgroups of charter schools by race, economic status, and pre-admission achievement were based on whether the charter 


schools’ students were above or below the median value of that characteristic across all charter school sites in the study. 
Thus, there were approximately equal numbers of charter schools in each of these subgroups. The number of students and 
sites varied by characteristic because of missing data: 2,873 students and 31 sites for economic status; 2,857 students and 
31 sites for race; and 2,853 students and 30 sites for pre-admission achievement. 


There were 21 non-urban and 10 urban charter schools. The sample for the urbanicity estimates included 2,873 students. 


Within each subgroup, the differences between lottery winners and lottery losers are not statistically significant at the .05 


level, two-tailed test. 
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Figure B.2. Subgroups of charter schools with different characteristics were similar in how they affected 
college degree completion or continued enrollment: Percentage of lottery winners and losers who 
completed college, by charter school characteristic 
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schools’ students were above or below the median value of that characteristic across all charter schools in the study. Thus, 
there were approximately equal numbers of charter schools in each of these subgroups. The number of students and schools 
varied by characteristic: 2,873 students and 31 charter school sites for economic status; 2,857 students and 31 charter 
school sites for race; and 2,853 students and 30 charter school sites for pre-admission achievement. 


There were 21 non-urban and 10 urban charter school sites. 2,873 students were included in the sample for the urbanicity 
estimates. 
Within each subgroup, the differences between lottery winners and lottery losers are not statistically significant at the .05 
level, two-tailed test. 
+The difference in impacts between charter schools serving more white students and charter schools serving more non-white students 
(9 percentage points) is significantly different from zero at the .05 level, two-tailed test. 
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Supporting Detail for the Relationship between Impacts on College Enrollment and 
Completion and Impacts on Student Achievement (Finding 2) 


The final set of tables and figures show the results of the analysis of whether charter schools’ 
impacts on middle school achievement were related to their impacts on college outcomes. The 
relationships between middle school achievement and college outcomes at the student level are 
presented in Table B.5, based on a regression analysis, and Table B.7, based on a calculation of simple 
correlation coefficients. The relationships between the success of charter middle schools in the study 
in improving middle school academic achievement and their success in improving college enrollment 
and completion are presented in Tables B.6 and B.8 and Figures B.3 and B.4. Although most estimates 
of charter schools’ impacts on middle school achievement were positively related to impacts on 
college outcomes, none of these regression coefficients (Table B.6) or correlation coefficients (Table 
B.8) were statistically significant. The lack of a strong relationship can be seen in the scatter plots 
shown in Figures B.3 and B.4. In each scatter plot, each data point represents a single charter school 
site in the study. The site’s position on the x-axis of the graph represents its estimated impact on 
students’ middle school achievement while its position on the y-axis represents its estimated impact 
on students’ college enrollment (Figure B.3) or completion (Figure B.4). Larger circles represent 
more precise estimates of a charter school site’s impact on students’ middle school achievement, and 
smaller circles represent less precise estimates. 
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Table B.5. Students with higher middle school academic achievement were predicted to have higher rates of college enrollment and completion 


XV] nays 1e-10 Mollni-ya=\alec-Mi amore) |(=\-<-M=1a)ce)|lnat=lalme-]acemexe)pale)(-ya(o)alm(e)ge) ele) au(e)amoym-1a0(0(-\a1t-) Mw -d\V(-lal-Me)ai-M-ir-]aver-1c0| 
deviation increase in middle school achievement 


One standard deviation increase in One standard deviation increase in 
Year 2 One standard deviation increase in average Year 2 math and reading 
reading scores Year 2 math scores FYore) gt) 


(1) (2) i) (4) (5) ) 


Predicted change in p-value Predicted change in p-value Predicted change in p-value 
postsecondary postsecondary postsecondary 

Postsecondary outcomes outcomes outcomes outcomes 
Ever enrolled in any postsecondary institution 0.08** 0.000 0.07** 0.000 0.09** 0.000 
Obtained a certificate, associate’s degree, or 

bachelor’s degree, or currently enrolled 0.14** 0.000 0.11** 0.000 0.15** 0.000 
Ever enrolled in a four-year college 0.16** 0.000 0.13** 0.000 0.17** 0.000 
Number of students 1,913 1,913 1,908 1,908 1,919 1,919 

Note: These estimates were calculated by regressing each postsecondary outcome on each middle school achievement outcome, using linear probability models with robust 


standard errors and weights to account for different selection probability of each student. The reported estimates are a student’s predicted change in postsecondary 
outcome (as a proportion), given a one standard deviation increase in student middle school achievement. 


The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or impact was equal to 0. 
*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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Table B.6. Estimated impacts of charter schools on middle school achievement did not significantly predict estimated impacts on college 
enrollment and completion 


tid Taare 1k=0 Me lla=1c=)ale-Ml amore) |(=\-1-M=1a]co)|tat=vanmr-Vacemexe)iale)(-14(0) aM lan] e-\e1tom (e) ge) eleau(e)ameym-1ue(e(>/a1t-) Me -1)V(-1 a= Me)al =) 
standard deviation increase in impacts on middle school achievement 


One standard deviation increase in OJatewoir-laver-laeme(=\7t-\(olaMlalerest- yom 
Year 2 One standard deviation increase in average Year 2 math and reading 
reading score impacts Year 2 math score impacts score impacts 


e) (2) 3) C) (3) (3) 


Predicted change in pvalue Predicted change in p-value Predicted change in p-value 
postsecondary postsecondary postsecondary 

Postsecondary impacts impacts impacts Tan) exe Xe} 
Ever enrolled in any postsecondary institution 0.04 0.777 0.00 0.988 0.01 0.957 
Obtained a certificate, associate’s degree or 

bachelor’s degree, or currently enrolled 0.10 0.533 0.07 0.531 0.10 0.495 
Ever enrolled in a four-year college -0.05 0.741 0.00 0.974 -0.02 0.894 
Number of students 1,913 1,913 1,913 1,913 1,913 1,913 
Number of sites 28 28 28 28 28 28 

Note: These estimates were calculated by regressing the site-level postsecondary impact estimates on each site-level middle school achievement impact, using linear 


probability models with robust standard errors. The reported estimates are the predicted difference in postsecondary impacts (as a proportion), given a one standard 
deviation increase in student middle school achievement impacts. 


The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or impact was equal to O. 
The estimates are not statistically significant at the .05 level, two-tailed test. 
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Table B.7. Students’ middle school academic achievement was positively correlated with college 
enrollment and completion 


Correlation between students’ middle school achievement and their 
college enrollment and completion 


3) 24) 1) 


Year 2 reading scores Year 2 math Year 2 math and 

Postsecondary outcomes scores reading scores 
Ever enrolled in any postsecondary institution 0.17** 0.16** 0.18** 
Obtained a certificate, associate’s degree or 

bachelor’s degree, or currently enrolled 0.26** 0.24** 0.27** 
Ever enrolled in a four-year college 0.29** 0.29** 0.32** 
Number of students 1,913 1,908 1,919 

Note: These estimates are the Pearson correlations of the student-level postsecondary outcomes and the student-level 


middle school achievement outcomes. 
*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 


Table B.8. Charter school impacts on middle school academic achievement were not significantly 
correlated with impacts on college enrollment and completion 


Correlation between impacts on middle school achievement and 
Hina) oxe\eitome)amexe)|(=1-4-M-1alce)iinal-yaimr=laremecolenye)(-1ale)a 


(1) 2) \)) 


Year 2 reading score Year 2 math score Year 2 math and 

Postsecondary impacts impacts impacts reading score impacts 
Ever enrolled in any postsecondary institution 0.06 0.00 0.01 
Obtained a certificate, associate’s degree or 

bachelor’s degree, or currently enrolled 0.14 0.15 0.16 
Ever enrolled in a four-year college -0.08 0.01 -0.04 
Number of sites 28 28 28 

Note: These estimates are the Pearson correlations of the site-level postsecondary impact estimates and the site-level 


middle school achievement impacts. 
The correlations are not statistically significant at the .05 level, two-tailed test. 
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Figure B.3. Scatter plots of charter school impacts on college enrollment versus impacts on middle 
school achievement 
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Each circle in a graph represents the estimated impact of a single charter school site on students’ middle school 
achievement (x-axis) and college enrollment (y-axis). Each dotted line represents the correlation between the impacts 
on achievement and the impacts on college outcomes, estimated with a linear regression model. Larger circles 
represent more precise estimates of a charter school site’s impact on students’ middle school achievement, and 
smaller circles represent less precise estimates. (The area of each circle is proportional to the inverse of the standard 
error of that school’s estimated impact on middle school achievement.) 
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Figure B.4. Scatter plots of charter school impacts on college completion versus impacts on middle 
school achievement 
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Note: Each circle in a graph represents the estimated impact of a single charter school site on students’ middle school 
achievement (x-axis) and college completion (y-axis). Each dotted line represents the correlation between the impacts 
on achievement and the impacts on college outcomes, estimated with a linear regression model. Larger circles 
represent more precise estimates of a charter school site’s impact on students’ middle school achievement, and 
smaller circles represent less precise estimates. (The area of each circle is proportional to the inverse of the standard 
error of that school’s estimated impact on middle school achievement.) 
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APPENDIX C: SENSITIVITY ANALYSES 


This appendix describes the analyses that were conducted to test whether the main findings were 
sensitive to the method or approach that was used to produce the findings. In other words, these 
sensitivity analyses examined whether the results of the study would change if a different method was 
used. The study team conducted four types of sensitivity analyses: (1) changing the sample of students 
used to calculate the impacts on college outcomes, (2) changing how the impacts for individual charter 
middle schools in the study were combined to calculate the overall impact of the charter middle 
schools, (3) using different methods for estimating the impacts of charter schools, and (4) improving 
the precision of the correlational estimates. The results of the sensitivity analyses were consistent with 
the main findings, meaning that the findings were similar, regardless of the approach used. 


Estimating impacts on college outcomes with the smaller sample of students used to 
estimate impacts on student achievement in the original study 


This analysis was conducted to ensure that any differences in the estimates of charter schools’ 
long-term effects on college outcomes (Findings | and 2) and their short-term effects on achievement 
(Gleason et al., 2010) were not caused by the use of different student samples to calculate each set of 
impacts. The impacts on college outcomes described under Findings 1 and 2 were based on almost 
all students for whom college information was available** — a sample of 2,873 students or 99 percent 
of all 2,904 students who went through lotteries at the study charter middle schools. This sample was 
substantially larger than the sample used to estimate the impacts of charter schools on students’ 
middle school achievement in the original study (Gleason et al., 2010). To determine the sensitivity 
of Findings 1 and 2 to the students included in the analysis, the study team estimated charter schools’ 
impacts on college enrollment and completion again but using the same group of students included 
in the original study sample—those with valid achievement test scores.”> Like the main results, none 
of the main impact estimates (Table C.1) or the differences across subgroups (Table C.2) were 
statistically significant. 


Estimating impacts with all students pooled together rather than estimating them 
separately by school and then combining 


The average impact of admission to a charter middle school could vary depending on how it is 
calculated. The impacts on college outcomes presented under Finding | and Finding 2 were estimated 
separately by charter school site, and then combined by taking a simple average (mean value) of these 
site-specific impact estimates (with each site weighted equally in the calculation). To assess the 
sensitivity of the results, an alternative approach was used that first combined (“pooled”) students 
from all sites and then calculated a single overall pooled impact estimate, rather than separate 
estimates for each charter school site. The pooled models included the same covariates as the main 
models that produced the estimates described in Findings 1 and 2. In practical terms, this pooled 
approach gives greater weight in the calculation of overall charter school impacts to those sites with 
more students in the analysis. The alternative results (columns | and 2 of Table C.3) were consistent 
with the main results: none of these impact estimates were statistically significant. 


Estimating impacts on binary outcomes using logit and probit models 


Most of the outcomes examined in this brief are “binary” in that they have two possible values — 
such as ever enrolled in college or not — and impacts for binary outcomes can be estimated in different 
ways that could conceivably produce different results. In the main analyses, ordinary least squares 
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(OLS) regression models (referred to as linear probability models when estimated with binary 
outcome variables) were used to estimate impacts because the results from linear probability models 
are easier to interpret than results based on other approaches. To ensure the study findings can be 
used with confidence, two different alternative approaches were used to reestimate the impacts of 
charter schools on the binary outcome variables (called logit and probit models) (columns 3—6 of 
Table C.3). The small number of students in each charter school site made it impossible to use the 
logit and probit models when calculating impacts separately by charter school site and averaging 
those site estimates together (the study’s main approach). Instead the logit and probit models build 
on the pooled sample of students approach discussed above. Each of the logit and probit models also 
include the same covariates as the original models. As reported above, the pooled approach with OLS 
for the binary outcomes produced results similar to the main findings. Similarly, the logit and probit 
models with the pooled sample approach also produced results similar to the main findings: none of 
these alternate estimates of the impacts of charter schools on key outcomes were statistically 
significant. For comparability across the OLS and probit and logit models, all of the findings in Table 
C.3 are presented as marginal impacts. 


Estimating correlations between charter schools’ impacts on student achievement and 
college outcomes using weighted least squares models and larger sample to improve 
precision 


It is possible that the study found no correlation between charter schools’ impacts on middle 
school achievement and impacts and college outcomes because the estimates of impacts of individual 
charter schools were imprecise. In other words, the student samples may have been too small to allow 
us to estimate the impacts of each school with much certainty. The study conducted two sensitivity 
tests to improve precision in the correlational analysis: (1) giving more weight in the calculation of 
the correlation to the more precisely estimated charter school impacts than to the less precisely 
estimated impacts, and (2) increasing the sample size. 


First, the correlations between middle school achievement impacts and college impacts were 
reestimated using a weighted least squares method, with the weights based on the precision of the 
school’s estimated achievement impacts. This approach potentially provides more precise estimates 
of the correlations between a school’s impacts on achievement and impacts on college outcomes. And 
because it gives different weight to different charter schools in the analysis, this estimated correlation 
itself could be different. Although the weighted least squares estimates do provide a more precise 
estimate of the correlation between impacts on achievement and impacts on college outcomes, none 
of the estimates were statistically significant (Table C.5), as in the main analysis. 


Second, to maximize the sample size used to conduct the correlational analysis between charter 
schools’ impacts on middle school achievement and college outcomes, impacts were estimated 
separately for each lottery in each school. Each of these estimated impacts was included as a separate 
observation in the analysis of correlations between impacts on achievement and attainment. For 
example, if a charter school conducted a lottery in 2005 for the 2005-2006 school year and in 2006 
for the 2006-2007 school year, the school contributed two separate impact estimates as observations 
to the analysis. The same sample of students (2,356) and charter sites was used to separately estimate 
middle school achievement and college enrollment and completion impacts. Then, the correlations 
between the impacts were estimated again. Like the main analyses, none of the correlations were 
statistically significant. 
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Table C.1. Charter school impacts on college enrollment and completion, using smaller sample of 
students used to estimate achievement impacts in original study 


Taal excleumeymr-(olnn ie) Meyni=le Tan) exs\oumeyme-1an=1aet=]aver:) 


ie) 29) i) (4) to) ) 


Mean, Mean, Difference p-value Adjusted p-value 
lottery lottery (impact Tan) exe\e9 
Outcome winners losers estimate) estimate 


Primary outcomes 

Ever enrolled in any postsecondary 0.72 0.71 0.01 0.662 0.00 0.941 
institution 

Obtained a certificate, associate’s degree, 0.48 0.48 0.00 0.898 -0.01 0.834 
or bachelor’s degree, or currently 
enrolled 


Exploratory outcomes related to college 
enrollment 


Ever enrolled in postsecondary institution 
2-year college 0.36 0.36 0.00 0.936 -0.01 0.869 
4-year college 0.56 0.54 0.02 0.393 0.02 0.694 


Type of institution where currently or most 
recently enrolled 


Public 0.59 0.58 0.01 0.778 0.00 0.951 

Private nonprofit 0.12 0.12 0.00 0.912 0.00 0.909 

Private for-profit 0.01 0.01 0.00 0.870 0.00 0.919 

Selectivity of postsecondary institution 

Highly selective (1-25% acceptance rate) 0.03 0.02 0.01 0.273 0.01 0.222 

Selective (26-50% acceptance rate) 0.15 0.14 0.00 0.826 -0.01 0.814 

Somewhat selective (51-75% acceptance 0.25 0.214 0.04 0.098 0.04 0.245 
rate) 

Less selective (76-100% acceptance rate) 0.30 0.33 -0.04 0.155 -0.06 0.225 

Dual enrollment participation 0.10 0.10 0.01 0.750 0.01 0.702 

Exploratory outcomes related to college 

completion 

Graduated from postsecondary institution@ 

Associate’s degree (2-year college) 0.03 0.04 -0.01 0.456 -0.01 0.585 

Bachelor’s degree (4-year college) 0.28 0.30 -0.02 0.527 -0.03 0.429 

Graduated with bachelor’s degree within 4 0.25 0.26 -0.01 0.673 -0.02 0.574 


years of projected high school graduation 


Graduated or currently enrolled 


Graduated with associate’s degree or 0.06 0.05 0.01 0.438 0.02 0.374 
currently enrolled in 2-year college 
Graduated with bachelor’s degree or 0.41 0.414 0.00 0.887 -0.01 0.801 


currently enrolled in 4-year college 


Number of years completed 


At 2-year college 0.41 0.40 0.01 0.819 0.00 0.986 
At 4-year college 1.95 1.93 0.02 0.832 -0.04 0.844 
At any postsecondary institution 2.51 2.48 0.02 0.840 -0.05 0.803 
Middle school Year 2 achievement? 

Reading 0.314 0.38 -0.07 0.032+ -0.08 0.117 
Math 0.32 0.38 -0.06 0.136 -0.08 0.202 
Number of students 1,169 744 1,913 1,905 

Number of sites 28 28 

Note: The estimates presented in this table use nearly the same sample as the original study estimates. The original study 


sample included 1,920 students in the Year 2 estimates, including 1,172 in the lottery winners group and 748 in the 
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lottery losers group. The estimates shown here have seven fewer students because these students did not have dates 
of birth and therefore were omitted from the NSC data request and the analysis. 


In both the lottery winners and lottery losers groups, 3 percent of students who attended a postsecondary institution 
opted out of sharing their data. In the student-level postsecondary data file, these students are indistinguishable from 
the students who never enrolled in a postsecondary institution, and are counted as if they never enrolled. Therefore, 
estimates presented in this table underestimate overall enrollment. 


Means are estimated at the site level and then averaged across all 29 sites in the study. The estimates are weighted 
according to each student’s probability of assignment to the lottery winner or lottery loser group. 


The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or 
impact was equal to 0. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled 
immediately after high school and for whom we have at least four years of additional data to measure college completion 
(n = 1,818 for the impact of admission offer estimates and n = 1,810 for the impact of attendance estimates). 


>The achievement impact estimates are from Table IV.1 of Gleason et al. (2010). 
*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
+Difference significantly different from zero at the .05 level, two-tailed test. 
++Difference significantly different from zero at the .01 level, two-tailed test. 
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Table C.2. Impacts on postsecondary enrollment and completion, by charter school characteristics, using smaller sample of students used to 
estimate achievement impacts in original study 


Percentage eligible for free or reduced- 
price meals Percentage of white students Mean pre-lottery achievement Urbanicity 


ce) 29) ()) (4) (3) ()) 0) (2) iC) (eo) (11) (12) 


Sites serving Sites serving Diff- Sites Sites Diff- Sites Sites Diff- Sites Sites in Diff- 
more fewer erence serving serving erence |servinglow- serving erence | outside of large erence 
economically economically rate) (=) more white achieving high- large urban 
advantaged advantaged non-white students students achieving urban areas 
Outcome students students students students Eliccky 


Primary outcomes 


Ever enrolled in any postsecondary -0.01 0.03 0.04 0.04 -0.03 -0.08 0.01 0.00 -0.01 0.01 0.00 -0.02 
institution 
Obtained a certificate, associate’s 0.00 -0.01 -0.01 0.00 0.00 0.00 0.00 0.00 0.00 -0.01 0.04 0.02 


degree, or bachelor’s degree, 
or currently enrolled 


Exploratory outcomes related to 
college enrollment 


Ever enrolled in postsecondary 


institution 
2-year college -0.04 0.03 0.07 0.04 -0.06 -0.10~ 0.02 -0.03 -0.05 -0.02 0.02 0.05 
4-year college 0.01 0.02 0.01 0.04 0.00 -0.04 0.00 0.04 0.04 0.02 0.01 -0.01 


Type of institution where currently 
or most recently enrolled 


Public 0.01 0.01 0.00 0.02 0.00 -0.02 0.02 0.00 -0.02 0.04 0.00 -0.01 

Private nonprofit -0.02 0.01 0.03 0.02 -0.03 -0.06 -0.01 0.00 0.01 0.00 -0.01 -0.01 

Private for-profit 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 

Selectivity of postsecondary 

institution 

Highly selective (1-25% 0.00 0.02* 0.02 0.01* 0.00 -0.02 0.00 0.01 0.01 0.00 0.01 0.01 
acceptance rate) 

Selective (26-50% acceptance 0.01 0.00 0.00 0.02 -0.02 -0.04 0.00 0.01 0.00 0.00 0.01 0.01 
rate) 

Somewhat selective (51-75% 0.02 0.05~ 0.03 0.03 0.04 0.01 0.06* 0.01 -0.04 0.03 0.04 0.01 
acceptance rate) 

Less selective (76-100% -0.04 -0.04 0.00 -0.03 -0.05 -0.03 -0.05 -0.03 0.02 -0.02 -0.07 -0.05 
acceptance rate) 

Dual enrollment participation -0.01 0.03 0.05 0.01 0.00 -0.01 0.01 0.00 -0.01 0.02 -0.01 -0.03 


Exploratory outcomes related to 
college completion 


Graduated from postsecondary 
institution? 
Associate’s degree (2-year college) -0.01 0.00 0.01 0.00 -0.02 -0.02 0.00 -0.014 0.00 -0.01 0.00 0.02 
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Percentage eligible for free or reduced- 
price meals Percentage of white students Mean pre-lottery achievement 


e) 22) \)) (4) (5) ()) 0) (3) iC) 


Sites serving Sites serving Diff- Sites Sites Diff- Sites Sites Diff- 
rate) (=) fewer erence serving serving erence |servinglow- serving erence 
economically economically more more white achieving high- 
advantaged advantaged non-white students students achieving 
Outcome students students students students 
Bachelor’s degree (4-year college) 0.00 -0.05 -0.05 -0.03 -0.02 0.014 -0.06~ 0.014 0.07 
Graduated with bachelor’s degree 0.01 -0.05~ -0.07 -0.02 -0.01 0.01 -0.06* 0.03 0.09~ 
within 4 years of projected high 
school graduation 


Graduated or currently enrolled 


Graduated with associate’s degree -0.01 0.03~ 0.04~ 0.03* -0.02 -0.05~ 0.02 -0.01 -0.03 
or currently enrolled in 2-year 
college 
Graduated with bachelor’s degree 0.03 -0.03 -0.06 -0.01 0.01 0.02 -0.02 0.02 0.04 
or currently enrolled in 4-year 
college 
Number of years completed 
At 2-year college 0.03 -0.02 -0.05 0.04 -0.04 -0.08 0.06 -0.06 -0.12 
At 4-year college 0.08 -0.06 -0.14 -0.02 0.06 0.08 -0.04 0.07 0.14 
At any postsecondary institution 0.07 -0.09 -0.16 0.05 -0.06 -0.14 0.03 -0.04 -0.07 


Middle school Year 2 achievement’ 


53 


Urbanicity 


Ge) (14) (12) 
Sites Sites in Diff- 
outside of large erence 

large urban 

Viger 1a) Elcky 

areas 

-0.01 -0.06 -0.05 
0.00 -0.04 -0.04 


0.00 0.03 0.03 


0.00 -0.01 -0.01 


0.00 0.04 0.04 


0.09 -0.16 -0.25 
0.07 -0.15 -0.21 


Reading -0.11 0) 0.44 -0.08* -0.03 0.05 0.03 -0.15** = -0.17++ -0.08* -0.02 0.06 
Math -0.24** 0.18** 0.414++ 0.01 -0.11 -0.12 0.12* -0.21** = -0.33++ -0.14**  0.16* 0.30++ 
Number of students 817 1,096 1,913 1,235 678 1,913 1,025 888 1,913 1,275 638 1,913 
Number of sites 15 13 28 15 13 28 14 14 28 19 9 28 
Note: The estimates presented in this table use nearly the same sample as the original study estimates. The original study sample included 1,920 students. The estimates shown here have 


seven fewer students because these students did not have dates of birth and therefore were omitted from the NSC data request and the analysis. 


In both the lottery winners and lottery losers groups, 3 percent of students who attended a postsecondary institution opted out of sharing their data. In the student-level postsecondary 
data file, these students are indistinguishable from the students who never enrolled in a postsecondary institution, and are counted as if they never enrolled. Therefore, estimates presented 


in this table underestimate overall enrollment. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled immediately after high school and for whom we have at least four years 


of additional data to measure college completion (n = 1,818). 


>’The achievement impact estimates are from Tables G.11, G.14, G.15, and G.16 of Gleason et al. (2010). The reported sample sizes vary depending on the charter school characteristic. 


*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 

+Difference is significantly different from zero at the .05 level, two-tailed test. 
++Difference is significantly different from zero at the .01 level, two-tailed test. 

~Estimate or difference is significantly different from zero at the .10 level, two-tailed test. 
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Table C.3. Impact estimates using ordinary least squares (OLS), logit, and probit models for pooled sample of all students 


(1) \)) )) 


OLS impact Logit impact ace) e)imm fen) ex- (ee 
Postsecondary impacts estimate estimate estimate 


Primary outcomes 

Ever enrolled in any postsecondary 0.01 0.655 0.01 0.675 0.01 0.590 
institution 

Obtained a certificate, associate’s degree, 0.01 0.799 0.00 0.836 0.00 0.816 
or bachelor’s degree, or currently 
enrolled 


Exploratory outcomes related to college 
enrollment 


Ever enrolled in postsecondary institution 
2-year college -0.01 0.637 -0.01 0.648 -0.01 0.606 
4-year college 0.03 0.171 0.03 0.186 0.03 0.170 


Type of institution where currently or most 
recently enrolled 


Public 0.01 0.557 0.01 0.525 0.01 0.529 
Private nonprofit -0.01 0.590 0.00 0.739 -0.01 0.680 
Private for-profit¢ 0.00 0.514 0.914 0.767 
Selectivity of postsecondary institution 
Highly selective (1-25% acceptance rate) 0.01 0.112 0.01 0.219 0.01 0.324 
Selective (26-50% acceptance rate) 0.01 0.316 0.02 0.220 0.02 0.219 
Somewhat selective (51-75% acceptance 0.03 0.107 0.03 0.071 0.03 0.091 
rate) 
Less selective (76-100% acceptance rate) -0.04* 0.024 -0.04* 0.039 -0.04* 0.034 
Dual enrollment participation -0.01 0.593 -0.01 0.489 -0.01 0.408 
Exploratory outcomes related to college 
completion 
Graduated from postsecondary institution@ 
Associate’s degree (2-year college) -0.01 0.541 0.00 0.689 -0.01 0.443 
Bachelor’s degree (4-year college) -0.01 0.721 0.00 0.800 -0.01 0.764 
Graduated with bachelor’s degree within 4 0.00 0.819 0.00 0.967 0.00 0.930 
years of projected high school 
graduation 
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(1) i) )) 


OLS impact Logit impact ede) e)i a(n) ex= (ea 
Postsecondary impacts estimate estimate estimate 


Graduated or currently enrolled 


Graduated with associate’s degree or 0.01 0.622 0.01 0.563 0.01 0.577 
currently enrolled in 2-year college 
Graduated with bachelor’s degree or 0.01 0.525 0.01 0.531 0.01 0.547 


currently enrolled in 4-year college 


Number of years completed> 


At 2-year college -0.04 0.393 - - - - 
At 4-year college 0.08 0.375 - - - - 
At any postsecondary institution 0.03 0.754 - - - - 
Number of students 2,895 2,895 2,895 
Note: The impact estimates are the marginal effects. Therefore, the logit and probit estimates are directly comparable to the impact estimates from the OLS models. 


In both the lottery winners and lottery losers groups, 3 percent of students who attended a postsecondary institution opted out of sharing their data. In the student- 
level postsecondary data file, these students are indistinguishable from the students who never enrolled in a postsecondary institution, and are counted as if they 
never enrolled. Therefore, estimates presented in this table underestimate overall enrollment by approximately 3 percentage points. 


The estimates are weighted according to each student’s probability of winning the charter school lottery. 
The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or impact was equal to 0. 


aThe sample for the graduation outcomes is limited to students who are projected to have completed college if they enrolled immediately after high school and for whom we have 
at least four years of additional data to measure college completion (n = 2,511 for the impact of admission offer estimates and n = 2,466 for the impact of attendance estimates). 


>‘Because number of years completed is a continuous variable, we do not report results from the logit and probit models. 

cThe logit and probit models for attendance at private for-profit colleges did not converge because only about 1 percent of students attended these types of colleges. 
*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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Table C.4. Predicted difference in postsecondary impacts, given a one standard deviation increase in impacts on middle school academic 
achievement, using weighted least squares method 


Postsecondary impacts 


e) 


Predicted change in 
postsecondary 
impacts 


Oyatewoir-lavet-leeme(=\7t-\(eaMlalerest- yom 
Year 2 reading score impacts 


14) 


p-value 


)) 


Predicted change in 
postsecondary 
impacts 


Oyatew-ir-lavet-leome(=\7t-\(e aM lalerest- yom a 
Year 2 math score impacts 


C) 


p-value 


(3) 


Predicted change in 
postsecondary 
impacts 


One standard deviation increase in 
average Year 2 math and reading 
score impacts 


(3) 


p-value 


Ever enrolled in any postsecondary institution 0.16 0.140 0.04 0.537 0.10 0.291 
Obtained a certificate, associate’s degree, or 
bachelor’s degree, or currently enrolled 0.14 0.259 0.06 0.500 0.11 0.356 
Ever enrolled in a four-year college 0.14 0.313 0.12 0.186 0.16 0.195 
Number of students 2,356 2,356 2,356 2,356 2,365 2,365 
Number of sites 33 33 33 33 33 33 
Note: We calculated these estimates by using weighted least squares regressions, with analytic weights based on the precision of a school’s estimated achievement impacts. 


We report the predicted difference in postsecondary impacts (as a proportion), given a one standard deviation increase in student middle school achievement impacts. 
The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or impact was equal to 0. 
To maximize sample size, we treated sites that conducted more than one lottery as two separate sites. This increased the sample size from the main impact estimates 


by two sites. 
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Table C.5. Students with higher middle school academic achievement were predicted to have higher rates of college enrollment and completion 


One standard deviation increase in 
One standard deviation increase in One standard deviation increase in average Year 2 math and reading 
Year 2 reading scores Year 2 math scores Yor) (=1-) 


(1) i) tS) (3) 
Predicted change in Predicted change in Predicted change in p-value 
postsecondary postsecondary postsecondary 
Postsecondary outcomes outcomes outcomes outcomes 


Ever enrolled in any postsecondary 
institution 0.08** 0.000 0.06** 0.000 0.08** 0.000 

Obtained a certificate, associate’s degree, 
or bachelor’s degree, or currently 


enrolled 0.13** 0.000 0.10** 0.000 0.14** 0.000 

Ever enrolled in a four-year college 0.15** 0.000 0.12** 0.000 0.16** 0.000 

Number of students 2,356 2,356 2,346 2,346 2,365 2,365 
Note: These estimates were calculated by regressing each postsecondary outcome on each middle school achievement outcome, using linear probability models with robust 


standard errors and weights to account for different selection probability of each student. The reported estimates are a student’s predicted change in postsecondary 
outcome (as a proportion), given a one standard deviation increase in student middle school achievement. 


The p-value represents the probability of finding the estimated difference or impact if, in fact, the true difference or impact was equal to O. 
*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 
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Table C.6. Estimated impacts of charter schools on middle school achievement did not significantly predict estimated impacts on college 
enrollment and completion 


One standard deviation increase in 
One standard deviation increase in 


Postsecondary impacts 


Year 2 reading score impacts 


e) 


Predicted change in 


postsecondary 
Taal exe Xe) 


One standard deviation increase in 


Year 2 math score impacts 


)) 


Predicted change in 


postsecondary 
Tan) exe \ets) 


(4) 


p-value 


(3) 


Predicted change in 


postsecondary 
impacts 


average Year 2 math and reading 
score impacts 


(3) 


p-value 


Ever enrolled in any postsecondary 

institution 0.05 0.634 -0.01 0.862 0.014 0.962 
Obtained a certificate, associate’s degree 

or bachelor’s degree, or currently 

enrolled 0.14 0.429 0.06 0.532 0.10 0.466 
Ever enrolled in a four-year college 0.04 0.807 0.04 0.723 0.05 0.769 
Number of students 2,356 2,356 2,356 2,356 2,365 2,365 
Number of sites 33 33 33 33 33 33 

Note: These estimates were calculated by regressing the site-level postsecondary impact estimates on each site-level middle school achievement impact, using linear 


probability models with robust standard errors. The reported estimates are the predicted difference in postsecondary impacts (as a proportion), given a one standard 


deviation increase in student middle school achievement impacts. 
The p-value represents the probability of getting the estimated difference or impact if, in fact, the true difference or impact was equal to O. 


To maximize sample size, we treated sites that conducted more than one lottery as two separate sites. This increased the sample size from the main impact estimates 


by two sites. 


The estimates are not statistically significant at the .05 level, two-tailed test. 
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Table C.7. Students’ middle school academic achievement was positively correlated with college enrollment and completion 


al 2 3 
Postsecondary outcomes Year 2 aie scores Year 2 Ae scores Year 2 math eee FYore) (=) 
Ever enrolled in any postsecondary institution 0.16** 0.15** 0.17** 
Obtained a certificate, associate’s degree or bachelor’s degree, 
or currently enrolled 0.26** 0.24** 0.27** 
Ever enrolled in a four-year college 0.29** 0.29** 0.32** 
Number of students 2,356 2,346 2,365 
Note: These estimates are the Pearson correlations of the student-level postsecondary outcomes and the student-level middle school achievement outcomes. 


To maximize sample size, the sites that conducted more than one lottery were treated as two separate sites. This increased the sample size from the main impact 
estimates by two sites. 


*Significantly different from zero at the .05 level, two-tailed test. 
**Significantly different from zero at the .01 level, two-tailed test. 


Table C.8. Charter school impacts on middle school academic achievement were not significantly correlated with impacts on college enrollment 
and completion 


1 2 3 

Postsecondary impacts Year 2 ie a impacts Year2 wae impacts Year 2 math and ie score impacts 
Ever enrolled in any postsecondary institution 0.08 -0.03 0.014 
Obtained a certificate, associate’s degree or bachelor’s degree, 

or currently enrolled 0.15 0.11 0.14 
Ever enrolled in a four-year college 0.05 0.08 0.07 
Number of sites 33 33 33 

Note: These estimates are the Pearson correlations of the site-level postsecondary impact estimates and the site-level middle school achievement impacts. 


To maximize sample size, the sites that conducted more than one lottery were treated as two separate sites. This increased the sample size from the main impact 
estimates by two sites. 


The correlations are not statistically significant at the .05 level, two-tailed test. 
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ENDNOTES 


' Although growth has slowed recently, new charter schools continue to open each year. More than 300 new charter 
schools opened in fall 2017, and after accounting for schools that closed, there was a still nationwide increase of about 70 
charters (David & Hesla, 2018). 


? More recently, a Center for Research on Education Outcomes study (CREDO, 2017) focused mainly on the impacts of 
charter management organizations but also examined independent charter schools. That study found small but positive 
impacts of charter management organizations, on average, and impacts of independent charter schools that were close to 
Zero. 


3 In addition to the research described here, a recent meta-analysis (Hitt, McShane, & Wolf, 2018) argued that in the area 
of school choice research more generally (including research on school voucher programs), “achievement impact 
estimates appear to be almost entirely uncorrelated with attainment impacts.” 


4 Throughout this evaluation brief, college enrollment refers to students’ enrollment in any postsecondary institution, 
including four-year colleges and universities, two-year colleges, and other postsecondary programs such as technical 
institutes. Similarly, the completion of a program of study at these institutions through receipt of a certificate, degree, or 
other credential is referred to as college completion or receipt of a college degree. 


> See: Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final 
report (NCEE 2010-4029). Washington, DC: National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, U.S. Department of Education. 


® Although 36 charter schools participated in the study, the analyses include only 31 charter school sites. Most charter 
schools are their own site, but in five cases two charter schools were combined into a single charter school site because 
these pairs of charter schools in the study shared some common applicants—students who applied to both schools in the 
pair. When this happened, the study team treated each pair as a single site and considered a student to be a lottery 
winner if he or she was offered admission to either of the schools in the pair. 


7 Although admissions offers did not guarantee that students would decide to attend the charter schools in this study, most 
lottery winners (78 percent) attended charter schools and most lottery losers (78 percent) attended traditional public 
schools. 


8 Nine students (three lottery winners and six lottery losers) were missing dates of birth, which are required for the 
National Student Clearinghouse data request. In addition, charter school sites with fewer than five lottery losers were 
excluded to ensure impact estimates were reliable. This led to an additional 22 students being excluded. 


° Students in the study applied to charter schools for either the 2005—2006 or 2006-2007 school years and participated in 
school lotteries to enter into 4th, Sth, 6th, or 7th grade. Therefore, students would be expected to graduate from high 
school as early as 2011 and as late as 2015, with a majority of students (82 percent) expected to graduate in 2012 or 2013. 


'0 The study used a statistical approach called linear regression analysis to account for any possible small differences in 
the two groups’ characteristics just prior to the lottery. This analysis used sample weights to account for the fact that 
different students had different probabilities of being offered admission to the charter school, as described in Appendix 
A. 


"| To estimate the effect of attending a charter school, the study team used the standard approach for estimating the impact 
of the treatment on the treated (TOT) developed by Angrist et al. (1996). This approach also accounts for the fact that a 
small proportion of lottery losers attended a charter school. The estimated effect of attending a charter school is based on 
adjusting the study’s estimate of the effect of being offered admission to a charter school using information on the 
proportion of lottery winners and lottery losers who attend a charter school. 


'2 Because students were followed for a limited time after their expected high school graduation (three to seven academic 
years), some students may have still been on track to earn a degree or certificate when the data collection period ended. 
Therefore, the study measured whether a student had earned a degree or was still enrolled at the end of the data collection 
period in December 2017. 


‘3 The college enrollment rates of both lottery winners and losers could be as high as 72 percent. The National Student 
Clearinghouse (NSC) reports a count of the number of students in the data request file who were found in their records 
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but opted out of allowing the NSC to share their data. Three percent of students in each group enrolled in a postsecondary 
institution but opted not to share their data. 


'4 & larger percentage of lottery winners than lottery losers enrolled in four-year schools (55 versus 52 percent), whereas 
the reverse was true of two-year school enrollment (32 percent among lottery winners and 35 percent among lottery 
losers). The difference between lottery winners and losers was not statistically significant for either outcome individually. 
However, another way of thinking about these results is that charter schools caused about 3 percent of students who would 
have enrolled in a two-year school to enroll in a four-year school instead; that is, they caused a shift in enrollment from 
two-year to four-year schools. This shift was marginally significant (p = 0.097). This effect is noted in the report’s text 
because other studies, including Angrist et al. (2016), did not find effects of charter schools on college enrollment overall 
but did find a shift in enrollment from two-year to four-year colleges. 


'S The selectivity of each college was based on the reported percentage of students accepted for admission at each 
institution in IES’s Integrated Postsecondary Education Data System. 


‘6 While the study was not able to document the high school that students attended, the NSC provides data on the dates 
and institutions of all colleges students attended, so it was possible to estimate which students took college courses while 
still in high school. 


'7 The Every Student Succeeds Act (ESSA) requires states to develop accountability systems that report on student growth 
and English language and mathematics proficiency among other indicators. Title I, Part A provides funding for states to 
develop new college and career readiness assessments. Taken together, there is a clear assumption that achievement, as 
measured by academic assessments, should link to college readiness and therefore college outcomes. 


'8 On average, lottery winners and losers with high middle school test scores were more likely to enroll in college and 
obtain a degree than those with lower scores. Middle school achievement was a stronger predictor of a student’s college 
completion (correlation of 0.27) than whether the student enrolled in college at all (correlation of 0.18) (see Appendix B, 
Tables B.5 and B.7 for more detail). (A correlation of 0 would have indicated no relationship at all between students’ 
middle school test scores and their college outcomes, while a correlation of 1 would have indicated that middle school 
test scores were a perfect predictor of students’ long-term college enrollment and completion.) These correlations, which 
are substantially less than the maximum correlation of 1.0, suggest that other factors also contributed to these long-term 
outcomes. As a point of reference, the College Board (which administers the SAT) reported a correlation of 0.30 between 
SAT math scores and first-year college grade-point average (Shaw et al., 2016). 


'? Throughout this appendix, college enrollment refers to students’ enrollment in any postsecondary institution, including 
four-year colleges and universities, two-year colleges, and other postsecondary programs such as technical institutes. 
Similarly, the completion of a program of study at these institutions through receipt of a certificate, degree, or other 
credential is referred to as college completion or receipt of a college degree. 


0 The original study team collected data on participating charter middle schools in fall 2006 and fall 2007. The team 
collected data on other charter middle schools in operation at the time of the study in fall 2007. 


21 A few students who were initially on the waiting list but were later offered admission to a study charter school after the 
first few months of the school year were also included in the lottery losers group. 


2 This study used the same models as Gleason et al. (2010), which followed the approach described by Angrist et al. 
(1996). 


3 The achievement impacts are reported in student standard deviation units. For example, an impact of one unit would 
indicate that admission to a charter school led to an increase in student achievement of one standard deviation in the 
student test score distribution. Most charter schools in the study had estimated impacts on student achievement in math 
and reading that were much less than 1, typically between -0.40 and +0.40. 


24 Twenty-two students from one charter site had college outcome data but were dropped from the analysis because the 
lottery loser group had fewer than five students. Because each charter site contributes equally to the results, Gleason et 
al. (2010) determined that sites with fewer than five students cannot be estimated with precision and therefore should be 
excluded from the analysis. To be consistent with that study, this analysis also adopted that approach. 


25 This sensitivity analysis included seven fewer students than the analyses from the original study because these students 
were lacking birth dates in the data. College outcome data were not available for these students, because birth dates are 
required for the NSC data request. 
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